You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/01/17 00:58:11 UTC

[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.6.0) - Build # 1203 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1203/
Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at __randomizedtesting.SeedInfo.seed([A69718B89E6AB27C:A29F974B8CCF5D5D]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:428)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:425)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:695)




Build Log:
[...truncated 9411 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 471994 T1634 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1389914009282
   [junit4]   2> 471995 T1634 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 472006 T1634 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 472007 T1634 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 472024 T1635 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 472032 T1634 oasc.ZkTestServer.run start zk server on port:54465
   [junit4]   2> 472088 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 472114 T1636 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 473512 T1638 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d8a0627 name:ZooKeeperConnection Watcher:127.0.0.1:54465/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 473512 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 473514 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 473518 T1643 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60e8dd29 name:ZooKeeperConnection Watcher:127.0.0.1:54465 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 473518 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 473532 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 473537 T1645 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25592bea name:ZooKeeperConnection Watcher:127.0.0.1:54465 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 473537 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 473538 T1634 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 473555 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 473565 T1634 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 473574 T1634 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 473581 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 473593 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 473598 T1647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59b3ef14 name:ZooKeeperConnection Watcher:127.0.0.1:54465/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 473599 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 473604 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 473614 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 473629 T1638 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 473630 T1647 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 473632 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 473641 T1649 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29ba29b3 name:ZooKeeperConnection Watcher:127.0.0.1:54465/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 473642 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 473645 T1634 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 473654 T1634 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 473675 T1634 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 473685 T1634 oasc.Overseer.start Overseer (id=91089404519448580-127.0.0.1:54465_solr-n_0000000000) starting
   [junit4]   2> 473692 T1634 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 473706 T1634 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 473720 T1634 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 473733 T1651 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 473738 T1651 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 473741 T1650 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 473755 T1650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 473757 T1650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 473758 T1650 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 473758 T1650 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 473767 T1649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473770 T1647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 473771 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 474248 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 474280 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 474302 T1649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474304 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 474311 T1650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 474317 T1638 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 474318 T1647 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 474327 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 474329 T1649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474332 T1653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f94d0a3 name:ZooKeeperConnection Watcher:127.0.0.1:54465/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 474332 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 474340 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 474349 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 474364 T1653 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 474366 T1638 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 474374 T1649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474374 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 474384 T1650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 474385 T1650 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 474400 T1649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474407 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 474424 T1649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474444 T1649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474558 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 474559 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 474570 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 474596 T1638 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 474599 T1653 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 474915 T1634 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54465 54465
   [junit4]   2> 474955 T1634 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 474971 T1634 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
   [junit4]   2> 474974 T1634 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 474989 T1654 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 475091 T1634 oasc.ZkTestServer.run start zk server on port:54473
   [junit4]   2> 475092 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 475099 T1660 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47063b1a name:ZooKeeperConnection Watcher:127.0.0.1:54473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 475100 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 475101 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 475106 T1662 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ad6fc23 name:ZooKeeperConnection Watcher:127.0.0.1:54473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 475106 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 475190 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 475194 T1664 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12ce9896 name:ZooKeeperConnection Watcher:127.0.0.1:54473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 475194 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 475195 T1634 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 475207 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 475216 T1634 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 475224 T1634 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 475238 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 475249 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 475253 T1666 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67cc4e4d name:ZooKeeperConnection Watcher:127.0.0.1:54473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 475254 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 475258 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 475268 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 475279 T1660 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 475279 T1666 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 475282 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 475286 T1668 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b75e66f name:ZooKeeperConnection Watcher:127.0.0.1:54473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 475286 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 475290 T1634 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 475300 T1634 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 475319 T1634 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 475328 T1634 oasc.Overseer.start Overseer (id=91089404710158340-127.0.0.1:54473_solr-n_0000000000) starting
   [junit4]   2> 475335 T1634 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 475348 T1634 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 475360 T1634 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 475369 T1670 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 475373 T1670 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 475376 T1669 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 475388 T1669 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 475390 T1669 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 475390 T1669 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 475390 T1669 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 475399 T1668 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 475401 T1666 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 475402 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 475885 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 475950 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 475982 T1668 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 475987 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 475992 T1669 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 476002 T1660 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 476003 T1666 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 476258 T1634 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54473 54473
   [junit4]   2> 476343 T1634 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 476361 T1634 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
   [junit4]   2> 476362 T1634 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 476364 T1671 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 476464 T1634 oasc.ZkTestServer.run start zk server on port:54479
   [junit4]   2> 476466 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 476473 T1677 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22e4c071 name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 476474 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 476475 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 476480 T1679 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d10729 name:ZooKeeperConnection Watcher:127.0.0.1:54479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 476480 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 476561 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 476566 T1681 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@589dd21f name:ZooKeeperConnection Watcher:127.0.0.1:54479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 476567 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 476568 T1634 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 476582 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 476596 T1682 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 476597 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 476602 T1684 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fb3878 name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 476603 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 476607 T1686 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62385052 name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 476607 T1682 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 476611 T1634 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 476612 T1682 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 476626 T1682 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 476627 T1634 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 476638 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 476655 T1682 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 476656 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 476670 T1682 oasc.Overseer.start Overseer (id=91089404799418372-127.0.0.1:54479_solr-n_0000000000) starting
   [junit4]   2> 476671 T1684 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 476678 T1634 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 476679 T1682 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 476697 T1682 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 476709 T1682 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 476720 T1682 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 476721 T1688 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 476726 T1682 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 476730 T1690 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e2acdad name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 476730 T1682 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 476744 T1682 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 476753 T1682 oasc.Overseer.start Overseer (id=91089404799418373-127.0.0.1:54479_solr-n_0000000001) starting
   [junit4]   2> 476774 T1692 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 476778 T1692 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 476780 T1691 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 476791 T1691 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 476794 T1691 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 476794 T1691 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 476795 T1691 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 476808 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 476812 T1684 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 477196 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 477261 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 477300 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477312 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477312 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 477314 T1691 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 477350 T1691 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 477351 T1691 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 477355 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 477368 T1694 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ef32c0d name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 477369 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 477373 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477377 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 477386 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 477398 T1684 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 477399 T1694 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 477406 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 477407 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477413 T1691 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 477413 T1691 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 477426 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477433 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 477540 T1684 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 477541 T1694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 477639 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 477649 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477649 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477649 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477655 T1691 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 477657 T1691 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 477658 T1691 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 477658 T1691 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 477668 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477778 T1684 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 477779 T1694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 478152 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 478179 T1684 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 478180 T1694 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 478187 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 478187 T1694 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 478206 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478206 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478207 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478212 T1691 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 478222 T1690 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478222 T1688 oasc.OverseerCollectionProcessor.amILeader  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:270)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:270)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:307)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:163)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:695)
   [junit4]   2> 
   [junit4]   2> 478223 T1688 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=91089404799418372-127.0.0.1:54479_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 478223 T1688 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 478223 T1688 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:222)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:219)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:219)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:219)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:170)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:695)
   [junit4]   2> 
   [junit4]   2> 478224 T1688 oasc.OverseerCollectionProcessor.amILeader  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:270)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:270)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:307)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:177)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:695)
   [junit4]   2> 
   [junit4]   2> 478224 T1688 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=91089404799418372-127.0.0.1:54479_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 478231 T1687 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer work queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.element(DistributedQueue.java:131)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:325)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:105)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:695)
   [junit4]   2> 
   [junit4]   2> 478331 T1694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 478393 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 478399 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 478405 T1682 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 478406 T1696 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21855b4d name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 478406 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 478414 T1682 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 478423 T1698 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37dce58d name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 478424 T1682 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 478424 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 478445 T1682 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 478445 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 478463 T1682 oasc.Overseer.start Overseer (id=91089404799418376-127.0.0.1:54479_solr-n_0000000002) starting
   [junit4]   2> 478463 T1694 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 478468 T1696 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 478481 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 478490 T1700 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 478494 T1700 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 478497 T1699 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 478511 T1699 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 478519 T1699 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 478519 T1699 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 478525 T1694 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 478526 T1696 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 478536 T1696 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 478538 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478547 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 478565 T1699 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 478571 T1699 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 478572 T1699 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 478578 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478579 T1696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 478593 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 478597 T1702 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a8c6475 name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 478597 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 478604 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 478605 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478614 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 478626 T1702 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 478627 T1696 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 478633 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478633 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 478641 T1699 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 478641 T1699 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 478654 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478664 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 478762 T1702 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 478764 T1696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 478869 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 478879 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 478879 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478880 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478881 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478895 T1699 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 478901 T1699 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 478901 T1699 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 478916 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478926 T1702 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 478928 T1696 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 478933 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 478933 T1702 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 478953 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 478971 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479083 T1702 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 479139 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479155 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 479160 T1704 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32fcd3ee name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 479161 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 479165 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 479175 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 479187 T1702 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 479189 T1704 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 479197 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479197 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 479198 T1682 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 479198 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479200 T1698 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479216 T1682 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 479220 T1706 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6eaaf5a4 name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 479221 T1682 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 479241 T1682 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 479250 T1704 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 479250 T1702 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 479256 T1682 oasc.Overseer.start Overseer (id=91089404799418379-127.0.0.1:54479_solr-n_0000000003) starting
   [junit4]   2> 479259 T1704 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 479266 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 479285 T1708 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 479289 T1708 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 479292 T1707 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 479307 T1707 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479309 T1707 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 479309 T1707 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 479311 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 479316 T1710 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28139cd0 name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 479316 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 479320 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479322 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 479324 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 479334 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 479349 T1704 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 479350 T1710 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 479351 T1707 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479354 T1707 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 479355 T1707 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 479359 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479361 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 479384 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479390 T1707 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 479390 T1707 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 479394 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479400 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479507 T1710 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 479508 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 479601 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479609 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479610 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479610 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479610 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 479620 T1707 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479624 T1707 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 479624 T1707 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 479641 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479658 T1704 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 479660 T1710 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 479663 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479669 T1710 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 479684 T1684 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 479684 T1684 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 479688 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479701 T1706 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479809 T1710 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 479870 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479877 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 479887 T1712 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fe08d9 name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 479887 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 479892 T1682 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 479898 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 479900 T1682 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 479910 T1714 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35dd051f name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 479911 T1682 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 479920 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 479927 T1682 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 479938 T1712 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 479940 T1710 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 479943 T1682 oasc.Overseer.start Overseer (id=91089404799418382-127.0.0.1:54479_solr-n_0000000004) starting
   [junit4]   2> 479948 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 479965 T1716 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 479967 T1716 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 479975 T1715 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 479991 T1712 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 479993 T1710 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 479997 T1715 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 480003 T1715 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 480004 T1715 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 480005 T1712 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 480005 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 480005 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480043 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480044 T1712 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 480050 T1715 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 480053 T1715 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 480054 T1715 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 480053 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 480056 T1718 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b15947b name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 480056 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 480062 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 480078 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 480083 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480098 T1718 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 480100 T1712 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 480107 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480107 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 480113 T1715 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 480114 T1715 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 480149 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480158 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 480258 T1718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 480258 T1712 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 480365 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 480385 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480385 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 480385 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480387 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480403 T1715 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 480408 T1715 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 480409 T1715 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 480422 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480430 T1696 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 480431 T1696 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 480432 T1718 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 480434 T1712 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 480439 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 480439 T1718 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 480457 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480468 T1714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480581 T1718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 480647 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 480658 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 480666 T1720 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22ed101f name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 480667 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 480673 T1682 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 480676 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 480681 T1682 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 480691 T1722 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a37db6f name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 480691 T1682 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 480696 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 480705 T1699 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-0000000012
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:270)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:270)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:428)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:391)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:163)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:695)
   [junit4]   2> 
   [junit4]   2> 480722 T1718 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 480722 T1682 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 480723 T1720 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 480736 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 480736 T1682 oasc.Overseer.start Overseer (id=91089404799418385-127.0.0.1:54479_solr-n_0000000005) starting
   [junit4]   2> 480762 T1724 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 480768 T1724 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 480771 T1723 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 480780 T1720 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 480782 T1718 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 480788 T1723 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 480791 T1723 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 480791 T1723 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 480793 T1720 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 480795 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 480798 T1722 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480831 T1723 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 480832 T1722 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480833 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 480836 T1723 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 480836 T1723 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 480841 T1634 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 480845 T1726 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31977370 name:ZooKeeperConnection Watcher:127.0.0.1:54479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 480845 T1634 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 480850 T1634 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 480862 T1722 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480864 T1634 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 480875 T1726 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 480876 T1720 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 480881 T1722 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480881 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 480887 T1723 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 480887 T1723 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 480898 T1722 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 480907 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 481011 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 481012 T1726 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 481113 T1634 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 481121 T1722 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 481122 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 481122 T1722 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 481123 T1722 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 481130 T1723 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 481135 T1723 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 481135 T1723 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 481146 T1722 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 481158 T1726 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 481159 T1704 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 481160 T1704 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
  

[...truncated too long message...]

pter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:695)
   [junit4]   2> 
   [junit4]   2> 521585 T2056 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:654)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:649)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:649)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:649)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:875)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:226)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:187)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:695)
   [junit4]   2> 	
   [junit4]   2> 521592 T1634 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 521600 T2055 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 521611 T2055 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 521619 T2056 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1
   [junit4]   2> 521620 T2043 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 521625 T2049 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 521626 T2049 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 521627 T2049 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 521852 T1634 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54604 54604
   [junit4]   2> 521885 T1634 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=A69718B89E6AB27C -Dtests.slow=true -Dtests.locale=fi_FI -Dtests.timezone=US/Indiana-Starke -Dtests.file.encoding=UTF-8
   [junit4] ERROR   3.93s | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A69718B89E6AB27C:A29F974B8CCF5D5D]:0)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:428)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:425)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:695)
   [junit4]   2> 521903 T1634 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 521904 T1634 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 524906 T1634 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=fi_FI, timezone=US/Indiana-Starke
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Apple Inc. 1.6.0_65 (64-bit)/cpus=2,threads=1,free=39345088,total=232517632
   [junit4]   2> NOTE: All tests run in this JVM: [DateMathParserTest, HdfsUnloadDistributedZkTest, TestLFUCache, SortByFunctionTest, TestFieldTypeResource, BasicDistributedZkTest, TestReloadAndDeleteDocs, TestDistributedSearch, TestSuggestSpellingConverter, DocValuesMultiTest, CollectionsAPIDistributedZkTest, DirectUpdateHandlerTest, TestNumberUtils, DeleteShardTest, TestSchemaSimilarityResource, TestUpdate, CopyFieldTest, HdfsLockFactoryTest, OverseerTest]
   [junit4] Completed in 53.00s, 8 tests, 1 error <<< FAILURES!

[...truncated 1027 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1541 tests, 1 error, 39 ignored (5 assumptions)

Total time: 87 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1211 - Still Failing!

Posted by Mark Miller <ma...@gmail.com>.
I filed SOLR-5645.


On Sun, Jan 19, 2014 at 12:08 PM, Mark Miller <ma...@gmail.com> wrote:

> Well, I thought it was https://issues.apache.org/jira/browse/SOLR-4805,
> but it’s not that, it’s just similar.
>
> Some clean up that still needs to be done here. It does affect 4.6.1 as
> well.
>
> - Mark
>
> On Jan 19, 2014, at 12:02 PM, Mark Miller <ma...@gmail.com> wrote:
>
> Well, this isn’t good. It’s an old bug brought back to life - its trying
> to register the core in zk again on reload - this was previously fixed.
> Hope it’s not in 4.6.1.
>
> - Mark
>
> On Jan 18, 2014, at 11:16 PM, Policeman Jenkins Server <
> jenkins@thetaphi.de> wrote:
>
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1211/
> Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
>
> 1 tests failed.
> FAILED:
>  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
>
> Error Message:
> reloadcollection the collection time out:60s
>
> Stack Trace:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> reloadcollection the collection time out:60s
> at __randomizedtesting.SeedInfo.seed([A836D24B8F744B19:29D05C53F82B2B25]:0)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
> at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:765)
> at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:196)
> at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
> at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at java.lang.Thread.run(Thread.java:744)
>
>
>
>
> Build Log:
> [...truncated 10447 lines...]
>  [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
>  [junit4]   2> 1753662 T5108
> oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext
> system property: /
>  [junit4]   2> 1753672 T5108 oas.SolrTestCaseJ4.setUp ###Starting
> testDistribSearch
>  [junit4]   2> Creating dataDir:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1390103593043
>  [junit4]   2> 1753674 T5108 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>  [junit4]   2> 1753675 T5109 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
>  [junit4]   2> 1753776 T5108 oasc.ZkTestServer.run start zk server on
> port:54105
>  [junit4]   2> 1753778 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1753786 T5115 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@692bd0f2name:ZooKeeperConnection Watcher:
> 127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1753787 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1753788 T5108 oascc.SolrZkClient.makePath makePath: /solr
>  [junit4]   2> 1753800 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1753806 T5117 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@46284bcename:ZooKeeperConnection Watcher:
> 127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1753806 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1753807 T5108 oascc.SolrZkClient.makePath makePath:
> /collections/collection1
>  [junit4]   2> 1753816 T5108 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/shards
>  [junit4]   2> 1753824 T5108 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection
>  [junit4]   2> 1753831 T5108 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/shards
>  [junit4]   2> 1753836 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml
> to /configs/conf1/solrconfig.xml
>  [junit4]   2> 1753837 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.xml
>  [junit4]   2> 1753850 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml
> to /configs/conf1/schema.xml
>  [junit4]   2> 1753851 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/schema.xml
>  [junit4]   2> 1753861 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
> to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>  [junit4]   2> 1753862 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>  [junit4]   2> 1753872 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt
> to /configs/conf1/stopwords.txt
>  [junit4]   2> 1753873 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/stopwords.txt
>  [junit4]   2> 1753886 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt
> to /configs/conf1/protwords.txt
>  [junit4]   2> 1753887 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/protwords.txt
>  [junit4]   2> 1753898 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml
> to /configs/conf1/currency.xml
>  [junit4]   2> 1753899 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/currency.xml
>  [junit4]   2> 1753909 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml
> to /configs/conf1/enumsConfig.xml
>  [junit4]   2> 1753911 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/enumsConfig.xml
>  [junit4]   2> 1753921 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json
> to /configs/conf1/open-exchange-rates.json
>  [junit4]   2> 1753922 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/open-exchange-rates.json
>  [junit4]   2> 1753930 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
> to /configs/conf1/mapping-ISOLatin1Accent.txt
>  [junit4]   2> 1753931 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/mapping-ISOLatin1Accent.txt
>  [junit4]   2> 1753941 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt
> to /configs/conf1/old_synonyms.txt
>  [junit4]   2> 1753942 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/old_synonyms.txt
>  [junit4]   2> 1753953 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt
> to /configs/conf1/synonyms.txt
>  [junit4]   2> 1753954 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/synonyms.txt
>  [junit4]   2> Using old style solr.xml
>  [junit4]   2> 1753972 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1753978 T5119 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@218b21e5name:ZooKeeperConnection Watcher:
> 127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1753978 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1753979 T5108 oascc.SolrZkClient.makePath makePath: /solr
>  [junit4]   2> 1753986 T5110 oazs.NIOServerCnxn.doIO WARN caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x143a8a1805b0002, likely client has closed socket
>  [junit4]   2> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
>  [junit4]   2> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>  [junit4]   2> at java.lang.Thread.run(Thread.java:744)
>  [junit4]   2>
>  [junit4]   2> 1753990 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1753999 T5121 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@3cb1f3aname:ZooKeeperConnection Watcher:
> 127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1754000 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1754003 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml
> to /configs/conf2/solrconfig.xml
>  [junit4]   2> 1754005 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/solrconfig.xml
>  [junit4]   2> 1754018 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml
> to /configs/conf2/schema.xml
>  [junit4]   2> 1754021 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/schema.xml
>  [junit4]   2> 1754036 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
> to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>  [junit4]   2> 1754037 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>  [junit4]   2> 1754045 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt
> to /configs/conf2/stopwords.txt
>  [junit4]   2> 1754046 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/stopwords.txt
>  [junit4]   2> 1754061 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt
> to /configs/conf2/protwords.txt
>  [junit4]   2> 1754063 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/protwords.txt
>  [junit4]   2> 1754071 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml
> to /configs/conf2/currency.xml
>  [junit4]   2> 1754073 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/currency.xml
>  [junit4]   2> 1754086 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json
> to /configs/conf2/open-exchange-rates.json
>  [junit4]   2> 1754087 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/open-exchange-rates.json
>  [junit4]   2> 1754105 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
> to /configs/conf2/mapping-ISOLatin1Accent.txt
>  [junit4]   2> 1754106 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/mapping-ISOLatin1Accent.txt
>  [junit4]   2> 1754115 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt
> to /configs/conf2/old_synonyms.txt
>  [junit4]   2> 1754116 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/old_synonyms.txt
>  [junit4]   2> 1754128 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt
> to /configs/conf2/synonyms.txt
>  [junit4]   2> 1754129 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/synonyms.txt
>  [junit4]   2> 1754139 T5108 oasc.AbstractZkTestCase.putConfig put
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
> to /configs/conf2/elevate.xml
>  [junit4]   2> 1754140 T5108 oascc.SolrZkClient.makePath makePath:
> /configs/conf2/elevate.xml
>  [junit4]   2> 1754561 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>  [junit4]   2> 1754575 T5108 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:54110
>  [junit4]   2> 1754576 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>  [junit4]   2> 1754576 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>  [junit4]   2> 1754577 T5108 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521
>  [junit4]   2> 1754577 T5108 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/'
>  [junit4]   2> 1754614 T5108 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solr.xml
>  [junit4]   2> 1754697 T5108 oasc.CoreContainer.<init> New CoreContainer
> 725528504
>  [junit4]   2> 1754697 T5108 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/]
>  [junit4]   2> 1754699 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 120000
>  [junit4]   2> 1754700 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to: http://
>  [junit4]   2> 1754700 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>  [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>  [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>  [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>  [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>  [junit4]   2> 1754702 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>  [junit4]   2> 1754702 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>  [junit4]   2> 1754713 T5108 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>  [junit4]   2> 1754714 T5108 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>  [junit4]   2> 1754714 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>  [junit4]   2> 1754715 T5108 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:54105/solr
>  [junit4]   2> 1754715 T5108 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>  [junit4]   2> 1754716 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1754721 T5132 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@20826f36name:ZooKeeperConnection Watcher:
> 127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1754722 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1754728 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1754733 T5134 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@3ee56b7aname:ZooKeeperConnection Watcher:
> 127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1754733 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1754737 T5108 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
>  [junit4]   2> 1754748 T5108 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
>  [junit4]   2> 1754758 T5108 oascc.SolrZkClient.makePath makePath:
> /live_nodes
>  [junit4]   2> 1754764 T5108 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54110_
>  [junit4]   2> 1754766 T5108 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:54110_
>  [junit4]   2> 1754776 T5108 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
>  [junit4]   2> 1754783 T5108 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
>  [junit4]   2> 1754796 T5108 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
>  [junit4]   2> 1754802 T5108 oasc.Overseer.start Overseer
> (id=91101829074190341-127.0.0.1:54110_-n_0000000000) starting
>  [junit4]   2> 1754813 T5108 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
>  [junit4]   2> 1754826 T5136 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
>  [junit4]   2> 1754827 T5108 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
>  [junit4]   2> 1754829 T5136
> oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing
> overseer nodes
>  [junit4]   2> 1754835 T5108 oascc.SolrZkClient.makePath makePath:
> /aliases.json
>  [junit4]   2> 1754842 T5108
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>  [junit4]   2> 1754851 T5135 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
>  [junit4]   2> 1754860 T5108 oass.SolrDispatchFilter.init
> user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>  [junit4]   2> 1754861 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>  [junit4]   2> 1754862 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1754867 T5139 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5bbaf1afname:ZooKeeperConnection Watcher:
> 127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1754868 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1754871 T5108
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>  [junit4]   2> 1754879 T5108 oasc.ChaosMonkey.monkeyLog monkey: init -
> expire sessions:false cause connection loss:false
>  [junit4]   2> 1755261 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>  [junit4]   2> 1755265 T5108 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:54114
>  [junit4]   2> 1755268 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>  [junit4]   2> 1755268 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>  [junit4]   2> 1755268 T5108 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250
>  [junit4]   2> 1755269 T5108 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/'
>  [junit4]   2> 1755308 T5108 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solr.xml
>  [junit4]   2> 1755412 T5108 oasc.CoreContainer.<init> New CoreContainer
> 96924362
>  [junit4]   2> 1755412 T5108 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/]
>  [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 120000
>  [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to: http://
>  [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>  [junit4]   2> 1755416 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>  [junit4]   2> 1755416 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>  [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>  [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>  [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>  [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>  [junit4]   2> 1755429 T5108 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>  [junit4]   2> 1755429 T5108 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>  [junit4]   2> 1755430 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>  [junit4]   2> 1755430 T5108 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:54105/solr
>  [junit4]   2> 1755430 T5108 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>  [junit4]   2> 1755433 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1755438 T5150 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@486daebcname:ZooKeeperConnection Watcher:
> 127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1755439 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1755446 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1755451 T5152 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@4d708b92name:ZooKeeperConnection Watcher:
> 127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1755452 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1755471 T5108
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>  [junit4]   2> 1756488 T5108 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54114_
>  [junit4]   2> 1756493 T5108 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:54114_
>  [junit4]   2> 1756508 T5134 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
>  [junit4]   2> 1756508 T5139 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
>  [junit4]   2> 1756510 T5152 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
>  [junit4]   2> 1756538 T5108 oass.SolrDispatchFilter.init
> user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>  [junit4]   2> 1756539 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>  [junit4]   2> 1756946 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>  [junit4]   2> 1756951 T5108 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:54117
>  [junit4]   2> 1756954 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>  [junit4]   2> 1756954 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>  [junit4]   2> 1756954 T5108 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911
>  [junit4]   2> 1756955 T5108 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/'
>  [junit4]   2> 1756992 T5108 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/solr.xml
>  [junit4]   2> 1757060 T5108 oasc.CoreContainer.<init> New CoreContainer
> 1734112660
>  [junit4]   2> 1757061 T5108 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/]
>  [junit4]   2> 1757062 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 120000
>  [junit4]   2> 1757062 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to: http://
>  [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>  [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>  [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>  [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>  [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>  [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>  [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>  [junit4]   2> 1757074 T5108 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>  [junit4]   2> 1757074 T5108 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>  [junit4]   2> 1757075 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>  [junit4]   2> 1757075 T5108 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:54105/solr
>  [junit4]   2> 1757075 T5108 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>  [junit4]   2> 1757077 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1757085 T5164 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@672b7c8fname:ZooKeeperConnection Watcher:
> 127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1757086 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1757094 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1757099 T5166 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@81214c4name:ZooKeeperConnection Watcher:
> 127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1757100 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1757110 T5108
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>  [junit4]   2> 1758123 T5108 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54117_
>  [junit4]   2> 1758128 T5108 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:54117_
>  [junit4]   2> 1758152 T5152 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
>  [junit4]   2> 1758155 T5139 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
>  [junit4]   2> 1758154 T5134 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
>  [junit4]   2> 1758155 T5166 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
>  [junit4]   2> 1758174 T5108 oass.SolrDispatchFilter.init
> user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>  [junit4]   2> 1758175 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>  [junit4]   2> 1758552 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>  [junit4]   2> 1758557 T5108 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:54120
>  [junit4]   2> 1758559 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>  [junit4]   2> 1758560 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>  [junit4]   2> 1758560 T5108 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547
>  [junit4]   2> 1758561 T5108 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/'
>  [junit4]   2> 1758598 T5108 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solr.xml
>  [junit4]   2> 1758704 T5108 oasc.CoreContainer.<init> New CoreContainer
> 76332827
>  [junit4]   2> 1758705 T5108 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/]
>  [junit4]   2> 1758706 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 120000
>  [junit4]   2> 1758708 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to: http://
>  [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>  [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>  [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>  [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>  [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>  [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>  [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>  [junit4]   2> 1758721 T5108 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>  [junit4]   2> 1758721 T5108 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>  [junit4]   2> 1758722 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>  [junit4]   2> 1758722 T5108 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:54105/solr
>  [junit4]   2> 1758722 T5108 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>  [junit4]   2> 1758724 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1758728 T5178 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@345687f2name:ZooKeeperConnection Watcher:
> 127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1758730 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1758737 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1758743 T5180 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@fde9546name:ZooKeeperConnection Watcher:
> 127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1758744 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1758758 T5108
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>  [junit4]   2> 1759770 T5108 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54120_
>  [junit4]   2> 1759776 T5108 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:54120_
>  [junit4]   2> 1759791 T5166 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
>  [junit4]   2> 1759792 T5139 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
>  [junit4]   2> 1759800 T5180 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
>  [junit4]   2> 1759801 T5152 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
>  [junit4]   2> 1759802 T5134 oascc.ZkStateReader$3.process Updating live
> nodes... (4)
>  [junit4]   2> 1759829 T5108 oass.SolrDispatchFilter.init
> user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>  [junit4]   2> 1759830 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>  [junit4]   2> 1760202 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>  [junit4]   2> 1760208 T5108 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:54123
>  [junit4]   2> 1760210 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>  [junit4]   2> 1760211 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>  [junit4]   2> 1760212 T5108 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201
>  [junit4]   2> 1760212 T5108 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/'
>  [junit4]   2> 1760251 T5108 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solr.xml
>  [junit4]   2> 1760338 T5108 oasc.CoreContainer.<init> New CoreContainer
> 723834019
>  [junit4]   2> 1760339 T5108 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/]
>  [junit4]   2> 1760340 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 120000
>  [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to: http://
>  [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>  [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>  [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>  [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>  [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>  [junit4]   2> 1760343 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>  [junit4]   2> 1760343 T5108 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>  [junit4]   2> 1760353 T5108 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>  [junit4]   2> 1760354 T5108 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>  [junit4]   2> 1760354 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>  [junit4]   2> 1760355 T5108 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:54105/solr
>  [junit4]   2> 1760355 T5108 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>  [junit4]   2> 1760357 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1760361 T5192 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@7f042f75name:ZooKeeperConnection Watcher:
> 127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1760363 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1760370 T5108 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>  [junit4]   2> 1760375 T5194 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@3326075aname:ZooKeeperConnection Watcher:
> 127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>  [junit4]   2> 1760375 T5108 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>  [junit4]   2> 1760387 T5108
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>  [junit4]   2> 1761398 T5108 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54123_
>  [junit4]   2> 1761407 T5108 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:54123_
>  [junit4]   2> 1761415 T5180 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
>  [junit4]   2> 1761416 T5134 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
>  [junit4]   2> 1761417 T5194 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
>  [junit4]   2> 1761417 T5152 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
>  [junit4]   2> 1761416 T5139 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
>  [junit4]   2> 1761417 T5166 oascc.ZkStateReader$3.process Updating live
> nodes... (5)
>  [junit4]   2> 1762080 T5108 oass.SolrDispatchFilter.init
> user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>  [junit4]   2> 1762080 T5108 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>  [junit4]   2> 1762088 T5147 oasha.CollectionsHandler.handleCreateAction
> Creating Collection :
> router.field=myOwnField&version=2&name=solrj_collection&action=CREATE&wt=javabin&
> router.name
> =compositeId&numShards=2&collection.configName=conf1&replicationFactor=2
>  [junit4]   2> 1762091 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/collection-queue-work state: SyncConnected type
> NodeChildrenChanged
>  [junit4]   2> 1762094 T5136 oasc.OverseerCollectionProcessor.run Overseer
> Collection Processor: Get the message
> id:/overseer/collection-queue-work/qn-0000000000 message:{
>  [junit4]   2>   "operation":"createcollection",
>  [junit4]   2>   "fromApi":"true",
>  [junit4]   2>   "name":"solrj_collection",
>  [junit4]   2>   "replicationFactor":"2",
>  [junit4]   2>   "collection.configName":"conf1",
>  [junit4]   2>   "numShards":"2",
>  [junit4]   2>   "router.field":"myOwnField",
>  [junit4]   2>   "router.name":"compositeId"}
>  [junit4]   2> 1762094 T5136
> oasc.OverseerCollectionProcessor.processMessage WARN
> OverseerCollectionProcessor.processMessage : createcollection , {
>  [junit4]   2>   "operation":"createcollection",
>  [junit4]   2>   "fromApi":"true",
>  [junit4]   2>   "name":"solrj_collection",
>  [junit4]   2>   "replicationFactor":"2",
>  [junit4]   2>   "collection.configName":"conf1",
>  [junit4]   2>   "numShards":"2",
>  [junit4]   2>   "router.field":"myOwnField",
>  [junit4]   2>   "router.name":"compositeId"}
>  [junit4]   2> 1762095 T5136
> oasc.OverseerCollectionProcessor.createConfNode creating collections conf
> node /collections/solrj_collection
>  [junit4]   2> 1762095 T5136 oascc.SolrZkClient.makePath makePath:
> /collections/solrj_collection
>  [junit4]   2> 1762107 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1762112 T5135 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>  [junit4]   2> 1762113 T5135
> oasc.Overseer$ClusterStateUpdater.buildCollection building a new
> collection: solrj_collection
>  [junit4]   2> 1762114 T5135
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> solrj_collection with shards [shard1, shard2]
>  [junit4]   2> 1762121 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1762123 T5152 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762123 T5139 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762124 T5194 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762123 T5180 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762124 T5166 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762124 T5134 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762208 T5136
> oasc.OverseerCollectionProcessor.createCollection going to create cores
> replicas shardNames [shard1, shard2] , repFactor : 2
>  [junit4]   2> 1762209 T5136
> oasc.OverseerCollectionProcessor.createCollection Creating shard
> solrj_collection_shard1_replica1 as part of slice shard1 of collection
> solrj_collection on 127.0.0.1:54114_
>  [junit4]   2> 1762209 T5136
> oasc.OverseerCollectionProcessor.createCollection Creating shard
> solrj_collection_shard1_replica2 as part of slice shard1 of collection
> solrj_collection on 127.0.0.1:54120_
>  [junit4]   2> 1762210 T5136
> oasc.OverseerCollectionProcessor.createCollection Creating shard
> solrj_collection_shard2_replica1 as part of slice shard2 of collection
> solrj_collection on 127.0.0.1:54110_
>  [junit4]   2> 1762210 T5136
> oasc.OverseerCollectionProcessor.createCollection Creating shard
> solrj_collection_shard2_replica2 as part of slice shard2 of collection
> solrj_collection on 127.0.0.1:54123_
>  [junit4]   2> 1762222 T5148 oasc.ZkController.publish publishing
> core=solrj_collection_shard1_replica1 state=down
>  [junit4]   2> 1762223 T5126 oasc.ZkController.publish publishing
> core=solrj_collection_shard2_replica1 state=down
>  [junit4]   2> 1762224 T5174 oasc.ZkController.publish publishing
> core=solrj_collection_shard1_replica2 state=down
>  [junit4]   2> 1762224 T5190 oasc.ZkController.publish publishing
> core=solrj_collection_shard2_replica2 state=down
>  [junit4]   2> 1762227 T5148 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>  [junit4]   2> 1762228 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1762228 T5126 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>  [junit4]   2> 1762229 T5174 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>  [junit4]   2> 1762229 T5190 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>  [junit4]   2> 1762233 T5135 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>  [junit4]   2> 1762236 T5135 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=2 message={
>  [junit4]   2>   "operation":"state",
>  [junit4]   2>   "state":"down",
>  [junit4]   2>   "base_url":"http://127.0.0.1:54114",
>  [junit4]   2>   "core":"solrj_collection_shard1_replica1",
>  [junit4]   2>   "roles":null,
>  [junit4]   2>   "node_name":"127.0.0.1:54114_",
>  [junit4]   2>   "shard":"shard1",
>  [junit4]   2>   "shard_range":null,
>  [junit4]   2>   "shard_state":"active",
>  [junit4]   2>   "shard_parent":null,
>  [junit4]   2>   "collection":"solrj_collection",
>  [junit4]   2>   "numShards":"2",
>  [junit4]   2>   "core_node_name":null}
>  [junit4]   2> 1762243 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1762246 T5135 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=2 message={
>  [junit4]   2>   "operation":"state",
>  [junit4]   2>   "state":"down",
>  [junit4]   2>   "base_url":"http://127.0.0.1:54110",
>  [junit4]   2>   "core":"solrj_collection_shard2_replica1",
>  [junit4]   2>   "roles":null,
>  [junit4]   2>   "node_name":"127.0.0.1:54110_",
>  [junit4]   2>   "shard":"shard2",
>  [junit4]   2>   "shard_range":null,
>  [junit4]   2>   "shard_state":"active",
>  [junit4]   2>   "shard_parent":null,
>  [junit4]   2>   "collection":"solrj_collection",
>  [junit4]   2>   "numShards":"2",
>  [junit4]   2>   "core_node_name":null}
>  [junit4]   2> 1762254 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1762257 T5135 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=2 message={
>  [junit4]   2>   "operation":"state",
>  [junit4]   2>   "state":"down",
>  [junit4]   2>   "base_url":"http://127.0.0.1:54123",
>  [junit4]   2>   "core":"solrj_collection_shard2_replica2",
>  [junit4]   2>   "roles":null,
>  [junit4]   2>   "node_name":"127.0.0.1:54123_",
>  [junit4]   2>   "shard":"shard2",
>  [junit4]   2>   "shard_range":null,
>  [junit4]   2>   "shard_state":"active",
>  [junit4]   2>   "shard_parent":null,
>  [junit4]   2>   "collection":"solrj_collection",
>  [junit4]   2>   "numShards":"2",
>  [junit4]   2>   "core_node_name":null}
>  [junit4]   2> 1762265 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1762267 T5135 oasc.Overseer$ClusterStateUpdater.updateState
> Update state numShards=2 message={
>  [junit4]   2>   "operation":"state",
>  [junit4]   2>   "state":"down",
>  [junit4]   2>   "base_url":"http://127.0.0.1:54120",
>  [junit4]   2>   "core":"solrj_collection_shard1_replica2",
>  [junit4]   2>   "roles":null,
>  [junit4]   2>   "node_name":"127.0.0.1:54120_",
>  [junit4]   2>   "shard":"shard1",
>  [junit4]   2>   "shard_range":null,
>  [junit4]   2>   "shard_state":"active",
>  [junit4]   2>   "shard_parent":null,
>  [junit4]   2>   "collection":"solrj_collection",
>  [junit4]   2>   "numShards":"2",
>  [junit4]   2>   "core_node_name":null}
>  [junit4]   2> 1762276 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1762383 T5152 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762383 T5166 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762383 T5134 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762383 T5139 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762383 T5194 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1762383 T5180 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>  [junit4]   2> 1763229 T5148 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for solrj_collection_shard1_replica1
>  [junit4]   2> 1763230 T5148 oasc.SolrXMLCoresLocator.doPersist Persisted
> core descriptions to
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solr.xml
>  [junit4]   2> 1763231 T5148 oasc.CoreContainer.create Creating SolrCore
> 'solrj_collection_shard1_replica1' using instanceDir:
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1
>  [junit4]   2> 1763231 T5148 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:solrj_collection
>  [junit4]   2> 1763230 T5126 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for solrj_collection_shard2_replica1
>  [junit4]   2> 1763233 T5148 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>  [junit4]   2> 1763233 T5126 oasc.SolrXMLCoresLocator.doPersist Persisted
> core descriptions to
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solr.xml
>  [junit4]   2> 1763234 T5126 oasc.CoreContainer.create Creating SolrCore
> 'solrj_collection_shard2_replica1' using instanceDir:
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1
>  [junit4]   2> 1763234 T5126 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:solrj_collection
>  [junit4]   2> 1763231 T5190 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for solrj_collection_shard2_replica2
>  [junit4]   2> 1763231 T5174 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for solrj_collection_shard1_replica2
>  [junit4]   2> 1763236 T5126 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>  [junit4]   2> 1763237 T5126 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/solrj_collection
>  [junit4]   2> 1763236 T5190 oasc.SolrXMLCoresLocator.doPersist Persisted
> core descriptions to
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solr.xml
>  [junit4]   2> 1763238 T5190 oasc.CoreContainer.create Creating SolrCore
> 'solrj_collection_shard2_replica2' using instanceDir:
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2
>  [junit4]   2> 1763239 T5190 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:solrj_collection
>  [junit4]   2> 1763233 T5148 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/solrj_collection
>  [junit4]   2> 1763240 T5126 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/'
>  [junit4]   2> 1763237 T5174 oasc.SolrXMLCoresLocator.doPersist Persisted
> core descriptions to
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solr.xml
>  [junit4]   2> 1763243 T5174 oasc.CoreContainer.create Creating SolrCore
> 'solrj_collection_shard1_replica2' using instanceDir:
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2
>  [junit4]   2> 1763243 T5174 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:solrj_collection
>  [junit4]   2> 1763242 T5190 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>  [junit4]   2> 1763245 T5190 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/solrj_collection
>  [junit4]   2> 1763247 T5174 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>  [junit4]   2> 1763250 T5174 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/solrj_collection
>  [junit4]   2> 1763249 T5148 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/'
>  [junit4]   2> 1763269 T5190 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/'
>  [junit4]   2> 1763277 T5174 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/'
>  [junit4]   2> 1763420 T5148 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: LUCENE_47
>  [junit4]   2> 1763477 T5126 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: LUCENE_47
>  [junit4]   2> 1763482 T5190 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: LUCENE_47
>  [junit4]   2> 1763477 T5174 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: LUCENE_47
>  [junit4]   2> 1763684 T5174 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>  [junit4]   2> 1763688 T5174 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>  [junit4]   2> 1763689 T5126 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>  [junit4]   2> 1763703 T5174 oass.IndexSchema.readSchema
> [solrj_collection_shard1_replica2] Schema name=test
>  [junit4]   2> 1763718 T5126 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>  [junit4]   2> 1763735 T5126 oass.IndexSchema.readSchema
> [solrj_collection_shard2_replica1] Schema name=test
>  [junit4]   2> 1763767 T5148 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>  [junit4]   2> 1763798 T5190 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>  [junit4]   2> 1763809 T5148 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>  [junit4]   2> 1763813 T5190 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>  [junit4]   2> 1763838 T5190 oass.IndexSchema.readSchema
> [solrj_collection_shard2_replica2] Schema name=test
>  [junit4]   2> 1763843 T5148 oass.IndexSchema.readSchema
> [solrj_collection_shard1_replica1] Schema name=test
>  [junit4]   2> 1765449 T5126 oass.ByteField.init WARN ByteField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>  [junit4]   2> 1765457 T5126 oass.ShortField.init WARN ShortField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>  [junit4]   2> 1765458 T5126 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>  [junit4]   2> 1765534 T5126 oass.IndexSchema.readSchema default search
> field in schema is text
>  [junit4]   2> 1765545 T5126 oass.IndexSchema.readSchema unique key field:
> id
>  [junit4]   2> 1765568 T5148 oass.ByteField.init WARN ByteField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>  [junit4]   2> 1765568 T5148 oass.ShortField.init WARN ShortField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>  [junit4]   2> 1765599 T5174 oass.ByteField.init WARN ByteField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>  [junit4]   2> 1765604 T5174 oass.ShortField.init WARN ShortField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>  [junit4]   2> 1765605 T5174 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>  [junit4]   2> 1765612 T5174 oass.IndexSchema.readSchema default search
> field in schema is text
>  [junit4]   2> 1765601 T5148 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>  [junit4]   2> 1765635 T5174 oass.IndexSchema.readSchema unique key field:
> id
>  [junit4]   2> 1765649 T5148 oass.IndexSchema.readSchema default search
> field in schema is text
>  [junit4]   2> 1765665 T5148 oass.IndexSchema.readSchema unique key field:
> id
>  [junit4]   2> 1765675 T5126 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>  [junit4]   2> 1765685 T5126 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>  [junit4]   2> 1765693 T5126 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>  [junit4]   2> 1765698 T5126
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>  [junit4]   2> 1765699 T5126
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>  [junit4]   2> 1765699 T5126 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>  [junit4]   2> 1765704 T5126
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>  [junit4]   2> 1765705 T5126
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>  [junit4]   2> 1765705 T5126 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>  [junit4]   2> 1765706 T5126 oasc.SolrCore.<init>
> [solrj_collection_shard2_replica1] Opening new SolrCore at
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/,
> dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/
>  [junit4]   2> 1765707 T5126 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>  [junit4]   2> 1765706 T5190 oass.ByteField.init WARN ByteField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>  [junit4]   2> 1765712 T5174 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>  [junit4]   2> 1765713 T5190 oass.ShortField.init WARN ShortField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>  [junit4]   2> 1765711 T5126 oasc.CachingDirectoryFactory.get return new
> directory for
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data
>  [junit4]   2> 1765721 T5174 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>  [junit4]   2> 1765718 T5190 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>  [junit4]   2> 1765730 T5190 oass.IndexSchema.readSchema default search
> field in schema is text
>  [junit4]   2> 1765735 T5174 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>  [junit4]   2> 1765722 T5126 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index/
>  [junit4]   2> 1765739 T5126 oasc.SolrCore.initIndex WARN
> [solrj_collection_shard2_replica1] Solr index directory
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index'
> doesn't exist. Creating new index...
>  [junit4]   2> 1765739 T5126 oasc.CachingDirectoryFactory.get return new
> directory for
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index
>  [junit4]   2> 1765740 T5126 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
> minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>  [junit4]   2> 1765740 T5190 oass.IndexSchema.readSchema unique key field:
> id
>  [junit4]   2> 1765743 T5174
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>  [junit4]   2> 1765746 T5174
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>  [junit4]   2> 1765749 T5174 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>  [junit4]   2> 1765745 T5148 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>  [junit4]   2> 1765756 T5174
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>  [junit4]   2> 1765757 T5174
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>  [junit4]   2> 1765758 T5174 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>  [junit4]   2> 1765758 T5174 oasc.SolrCore.<init>
> [solrj_collection_shard1_replica2] Opening new SolrCore at
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/,
> dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/
>  [junit4]   2> 1765758 T5174 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>  [junit4]   2> 1765748 T5126 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>  [junit4]   2>
> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7477e0e2lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e4480e6
> ),segFN=segments_1,generation=1}
>  [junit4]   2> 1765760 T5126 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>  [junit4]   2> 1765761 T5174 oasc.CachingDirectoryFactory.get return new
> directory for
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data
>  [junit4]   2> 1765768 T5126 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>  [junit4]   2> 1765768 T5148 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>  [junit4]   2> 1765769 T5174 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index/
>  [junit4]   2> 1765770 T5174 oasc.SolrCore.initIndex WARN
> [solrj_collection_shard1_replica2] Solr index directory
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index'
> doesn't exist. Creating new index...
>  [junit4]   2> 1765771 T5174 oasc.CachingDirectoryFactory.get return new
> directory for
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index
>  [junit4]   2> 1765773 T5174 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
> minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>  [junit4]   2> 1765775 T5174 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>  [junit4]   2>
> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4a94a9c3lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cd26ede
> ),segFN=segments_1,generation=1}
>  [junit4]   2> 1765775 T5174 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>  [junit4]   2> 1765778 T5148 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>  [junit4]   2> 1765769 T5126 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>  [junit4]   2> 1765780 T5126 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>  [junit4]   2> 1765780 T5174 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>  [junit4]   2> 1765781 T5174 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>  [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>  [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>  [junit4]   2> 1765780 T5126 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>  [junit4]   2> 1765783 T5126 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>  [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>  [junit4]   2> 1765784 T5174 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>  [junit4]   2> 1765782 T5148
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>  [junit4]   2> 1765785 T5174 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>  [junit4]   2> 1765786 T5174 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>  [junit4]   2> 1765786 T5174 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>  [junit4]   2> 1765788 T5174 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>  [junit4]   2> 1765789 T5174 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>  [junit4]   2> 1765789 T5174 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>  [junit4]   2> 1765791 T5174 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>  [junit4]   2> 1765785 T5148
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>  [junit4]   2> 1765792 T5148 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>  [junit4]   2> 1765784 T5126 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>  [junit4]   2> 1765792 T5174 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>  [junit4]   2> 1765795 T5126 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>  [junit4]   2> 1765795 T5174 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>  [junit4]   2> 1765797 T5148
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>  [junit4]   2> 1765799 T5148
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>  [junit4]   2> 1765799 T5148 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>  [junit4]   2> 1765797 T5126 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>  [junit4]   2> 1765800 T5126 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>  [junit4]   2> 1765800 T5148 oasc.SolrCore.<init>
> [solrj_collection_shard1_replica1] Opening new SolrCore at
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/,
> dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/
>  [junit4]   2> 1765802 T5126 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>  [junit4]   2> 1765802 T5190 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>  [junit4]   2> 1765798 T5174 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>  [junit4]   2> 1765803 T5126 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>  [junit4]   2> 1765803 T5126 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>  [junit4]   2> 1765804 T5126 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>  [junit4]   2> 1765802 T5148 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>  [junit4]   2> 1765807 T5126 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>  [junit4]   2> 1765809 T5126 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>  [junit4]   2> 1765810 T5126 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>  [junit4]   2> 1765813 T5148 oasc.CachingDirectoryFactory.get return new
> directory for
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data
>  [junit4]   2> 1765814 T5148 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index/
>  [junit4]   2> 1765814 T5148 oasc.SolrCore.initIndex WARN
> [solrj_collection_shard1_replica1] Solr index directory
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index'
> doesn't exist. Creating new index...
>  [junit4]   2> 1765816 T5126 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>  [junit4]   2> 1765820 T5148 oasc.CachingDirectoryFactory.get return new
> directory for
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index
>  [junit4]   2> 1765820 T5148 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
> minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>  [junit4]   2> 1765817 T5174 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>  [junit4]   2> 1765823 T5190 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>  [junit4]   2> 1765823 T5126 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>  [junit4]   2> 1765825 T5126 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>  [junit4]   2> 1765826 T5126 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy:
> minMergeSize=0, mergeFactor=10, maxMergeSize=892100269,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
>  [junit4]   2> 1765827 T5126 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>  [junit4]   2>
> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7477e0e2lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e4480e6
> ),segFN=segments_1,generation=1}
>  [junit4]   2> 1765827 T5126 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>  [junit4]   2> 1765828 T5126 oass.SolrIndexSearcher.<init> Opening
> Searcher@190a2642 main
>  [junit4]   2> 1765829 T5148 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>  [junit4]   2>
> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@775d1479lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75a7b414
> ),segFN=segments_1,generation=1}
>  [junit4]   2> 1765830 T5148 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>  [junit4]   2> 1765834 T5200 oasc.SolrCore.registerSearcher
> [solrj_collection_shard2_replica1] Registered new searcher Searcher@190a2642main{StandardDirectoryReader(segments_1:1:nrt)}
>  [junit4]   2> 1765837 T5126 oasc.CoreContainer.registerCore registering
> core: solrj_collection_shard2_replica1
>  [junit4]   2> 1765837 T5126 oasc.ZkController.register Register replica -
> core:solrj_collection_shard2_replica1 address:http://127.0.0.1:54110collection:solrj_collection shard:shard2
>  [junit4]   2> 1765841 T5148 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>  [junit4]   2> 1765841 T5148 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>  [junit4]   2> 1765842 T5148 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>  [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>  [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>  [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>  [junit4]   2> 1765844 T5148 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>  [junit4]   2> 1765844 T5126 oascc.SolrZkClient.makePath makePath:
> /collections/solrj_collection/leader_elect/shard2/election
>  [junit4]   2> 1765841 T5190 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>  [junit4]   2> 1765845 T5148 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>  [junit4]   2> 1765846 T5148 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>  [junit4]   2> 1765842 T5174 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>  [junit4]   2> 1765848 T5174 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>  [junit4]   2> 1765847 T5148 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>  [junit4]   2> 1765851 T5174 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy:
> minMergeSize=0, mergeFactor=10, maxMergeSize=892100269,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
>  [junit4]   2> 1765854 T5174 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>  [junit4]   2>
> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4a94a9c3lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cd26ede
> ),segFN=segments_1,generation=1}
>  [junit4]   2> 1765854 T5174 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>  [junit4]   2> 1765855 T5174 oass.SolrIndexSearcher.<init> Opening
> Searcher@290c4ce6 main
>  [junit4]   2> 1765852 T5190
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>  [junit4]   2> 1765857 T5190
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>  [junit4]   2> 1765858 T5190 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>  [junit4]   2> 1765858 T5148 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>  [junit4]   2> 1765859 T5148 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>  [junit4]   2> 1765859 T5148 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>  [junit4]   2> 1765860 T5148 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>  [junit4]   2> 1765860 T5148 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>  [junit4]   2> 1765861 T5148 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>  [junit4]   2> 1765865 T5148 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>  [junit4]   2> 1765865 T5201 oasc.SolrCore.registerSearcher
> [solrj_collection_shard1_replica2] Registered new searcher Searcher@290c4ce6main{StandardDirectoryReader(segments_1:1:nrt)}
>  [junit4]   2> 1765869 T5174 oasc.CoreContainer.registerCore registering
> core: solrj_collection_shard1_replica2
>  [junit4]   2> 1765870 T5174 oasc.ZkController.register Register replica -
> core:solrj_collection_shard1_replica2 address:http://127.0.0.1:54120collection:solrj_collection shard:shard1
>  [junit4]   2> 1765872 T5190
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>  [junit4]   2> 1765873 T5190
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>  [junit4]   2> 1765873 T5190 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>  [junit4]   2> 1765874 T5190 oasc.SolrCore.<init>
> [solrj_collection_shard2_replica2] Opening new SolrCore at
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/,
> dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/
>  [junit4]   2> 1765875 T5190 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>  [junit4]   2> 1765876 T5174 oascc.SolrZkClient.makePath makePath:
> /collections/solrj_collection/leader_elect/shard1/election
>  [junit4]   2> 1765876 T5190 oasc.CachingDirectoryFactory.get return new
> directory for
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data
>  [junit4]   2> 1765878 T5190 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index/
>  [junit4]   2> 1765878 T5190 oasc.SolrCore.initIndex WARN
> [solrj_collection_shard2_replica2] Solr index directory
> './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index'
> doesn't exist. Creating new index...
>  [junit4]   2> 1765878 T5190 oasc.CachingDirectoryFactory.get return new
> directory for
> ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index
>  [junit4]   2> 1765879 T5190 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
> minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>  [junit4]   2> 1765879 T5148 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>  [junit4]   2> 1765880 T5148 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>  [junit4]   2> 1765881 T5148 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy:
> minMergeSize=0, mergeFactor=10, maxMergeSize=892100269,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
>  [junit4]   2> 1765883 T5148 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>  [junit4]   2>
> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@775d1479lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75a7b414
> ),segFN=segments_1,generation=1}
>  [junit4]   2> 1765885 T5148 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>  [junit4]   2> 1765886 T5148 oass.SolrIndexSearcher.<init> Opening
> Searcher@3dd43c40 main
>  [junit4]   2> 1765884 T5190 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>  [junit4]   2>
> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@40bfb0dflockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6da86f2c
> ),segFN=segments_1,generation=1}
>  [junit4]   2> 1765889 T5190 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>  [junit4]   2> 1765893 T5204 oasc.SolrCore.registerSearcher
> [solrj_collection_shard1_replica1] Registered new searcher Searcher@3dd43c40main{StandardDirectoryReader(segments_1:1:nrt)}
>  [junit4]   2> 1765896 T5148 oasc.CoreContainer.registerCore registering
> core: solrj_collection_shard1_replica1
>  [junit4]   2> 1765897 T5148 oasc.ZkController.register Register replica -
> core:solrj_collection_shard1_replica1 address:http://127.0.0.1:54114collection:solrj_collection shard:shard1
>  [junit4]   2> 1765896 T5190 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>  [junit4]   2> 1765899 T5190 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>  [junit4]   2> 1765896 T5126
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard2
>  [junit4]   2> 1765901 T5190 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>  [junit4]   2> 1765903 T5148 oascc.SolrZkClient.makePath makePath:
> /collections/solrj_collection/leader_elect/shard1/election
>  [junit4]   2> 1765909 T5190 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>  [junit4]   2> 1765911 T5190 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>  [junit4]   2> 1765912 T5190 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>  [junit4]   2> 1765913 T5190 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>  [junit4]   2> 1765913 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1765918 T5190 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>  [junit4]   2> 1765919 T5190 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>  [junit4]   2> 1765918 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1765920 T5134
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>  [junit4]   2> 1765921 T5190 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>  [junit4]   2> 1765922 T5190 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>  [junit4]   2> 1765923 T5190 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>  [junit4]   2> 1765924 T5190 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>  [junit4]   2> 1765925 T5126
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we
> see more replicas up for shard shard2: total=2 found=1 timeoutin=-8
>  [junit4]   2> 1765926 T5126
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for
> replicas to come up, but they are taking too long - assuming they won't
> come back till later
>  [junit4]   2> 1765927 T5126
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
>  [junit4]   2> ASYNC  NEW_CORE C3406 name=solrj_collection_shard2_replica1
> org.apache.solr.core.SolrCore@5da01731 url=
> http://127.0.0.1:54110/solrj_collection_shard2_replica1node=127.0.0.1:54110_
> C3406_STATE=coll:solrj_collection core:solrj_collection_shard2_replica1
> props:{state=down, base_url=http://127.0.0.1:54110,
> core=solrj_collection_shard2_replica1, node_name=127.0.0.1:54110_}
>  [junit4]   2> 1765929 T5126 C3406 P54110 oasc.SyncStrategy.sync Sync
> replicas to http://127.0.0.1:54110/solrj_collection_shard2_replica1/
>  [junit4]   2> 1765932 T5126 C3406 P54110 oasu.PeerSync.sync PeerSync:
> core=solrj_collection_shard2_replica1 url=http://127.0.0.1:54110 START
> replicas=[http://127.0.0.1:54123/solrj_collection_shard2_replica2/]
> nUpdates=100
>  [junit4]   2> 1765933 T5126 C3406 P54110 oasu.PeerSync.sync PeerSync:
> core=solrj_collection_shard2_replica1 url=http://127.0.0.1:54110 DONE.
>  We have no versions.  sync failed.
>  [junit4]   2> 1765934 T5126 C3406 P54110 oasc.SyncStrategy.syncReplicas
> Leader's attempt to sync with shard failed, moving to the next candidate
>  [junit4]   2> 1765935 T5126
> oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we
> have no versions - we can't sync in that case - we were active before, so
> become leader anyway
>  [junit4]   2> 1765936 T5126
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:54110/solrj_collection_shard2_replica1/ shard2
>  [junit4]   2> 1765938 T5126 oascc.SolrZkClient.makePath makePath:
> /collections/solrj_collection/leaders/shard2
>  [junit4]   2> 1765926 T5190 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>  [junit4]   2> 1765932 T5135 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>  [junit4]   2>
>
> [...truncated too long message...]
>
> ZkCmdExecutor.java:73)
>  [junit4]   2> at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>  [junit4]   2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>  [junit4]   2> at
> org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>  [junit4]   2> at
> org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>  [junit4]   2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>  [junit4]   2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>  [junit4]   2>
>  [junit4]   2> 1991623 T5194 oascc.ZkStateReader$3.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
>  [junit4]   2> 1991623 T5194 oasc.LeaderElector$1.process WARN
>  org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for
> /collections/awholynewcollection_2/leader_elect/shard1/election
>  [junit4]   2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>  [junit4]   2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>  [junit4]   2> at
> org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>  [junit4]   2> at
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>  [junit4]   2> at
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>  [junit4]   2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>  [junit4]   2> at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>  [junit4]   2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>  [junit4]   2> at
> org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>  [junit4]   2> at
> org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>  [junit4]   2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>  [junit4]   2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>  [junit4]   2>
>  [junit4]   2> 1991623 T5194 oasc.LeaderElector$1.process WARN
>  org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for
> /collections/awholynewcollection_0/leader_elect/shard3/election
>  [junit4]   2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>  [junit4]   2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>  [junit4]   2> at
> org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>  [junit4]   2> at
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>  [junit4]   2> at
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>  [junit4]   2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>  [junit4]   2> at
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>  [junit4]   2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>  [junit4]   2> at
> org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>  [junit4]   2> at
> org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>  [junit4]   2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>  [junit4]   2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>  [junit4]   2>
>  [junit4]   2> 1991804 T5180 oascc.ZkStateReader$3.process WARN ZooKeeper
> watch triggered, but Solr cannot talk to ZK
>  [junit4]   2> NOTE: test params are: codec=Lucene46:
> {range_facet_si=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136),
> range_facet_sl=PostingsFormat(name=Memory doPackFST= false),
> id=PostingsFormat(name=Memory doPackFST= false),
> a_si=Lucene41(blocksize=128), other_tl1=Lucene41(blocksize=128),
> text=PostingsFormat(name=MockRandom), multiDefault=Pulsing41(freqCutoff=1
> minBlockSize=29 maxBlockSize=136), intDefault=Lucene41(blocksize=128),
> timestamp=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136),
> a_t=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136),
> range_facet_l=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136),
> _version_=Lucene41(blocksize=128)}, docValues:{},
> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=de_LU,
> timezone=CET
>  [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51
> (64-bit)/cpus=2,threads=3,free=99625584,total=338677760
>  [junit4]   2> NOTE: All tests run in this JVM: [TestSolrJ,
> TestDistributedGrouping, TestFreeTextSuggestions,
> TestSchemaSimilarityResource, DirectUpdateHandlerTest,
> PreAnalyzedUpdateProcessorTest, TestCoreDiscovery,
> CoreMergeIndexesAdminHandlerTest, TestClassNameShortening, IndexSchemaTest,
> TestWordDelimiterFilterFactory, FullSolrCloudDistribCmdsTest,
> TestShardHandlerFactory, CircularListTest, TestRealTimeGet,
> SolrPluginUtilsTest, TestQueryUtils, TestExtendedDismaxParser,
> TestSchemaVersionResource, TestPostingsSolrHighlighter,
> UniqFieldsUpdateProcessorFactoryTest, MoreLikeThisHandlerTest,
> TestPhraseSuggestions, TestFuzzyAnalyzedSuggestions, UpdateParamsTest,
> TestInfoStreamLogging, BasicDistributedZk2Test, SyncSliceTest,
> TestPerFieldSimilarity, TestNRTOpen, BasicFunctionalityTest,
> TestAtomicUpdateErrorCases, SliceStateUpdateTest, JsonLoaderTest,
> TestMaxScoreQueryParser, BadIndexSchemaTest, PrimUtilsTest,
> TestMergePolicyConfig, BadCopyFieldTest, TestOmitPositions,
> TestRandomFaceting, RAMDirectoryFactoryTest, RequestHandlersTest,
> AliasIntegrationTest, HighlighterConfigTest, TestFileDictionaryLookup,
> TestLuceneMatchVersion, TestDefaultSearchFieldResource,
> TriLevelCompositeIdRoutingTest, UUIDFieldTest, TestSurroundQueryParser,
> StandardRequestHandlerTest, TestHashPartitioner, RegexBoostProcessorTest,
> TestBM25SimilarityFactory, SuggesterWFSTTest, DocumentBuilderTest,
> TestSolrDeletionPolicy2, SuggesterTest, OutputWriterTest, SimpleFacetsTest,
> FieldAnalysisRequestHandlerTest, TestFieldTypeResource,
> SolrIndexSplitterTest, TestPHPSerializedResponseWriter,
> SpellCheckComponentTest, UpdateRequestProcessorFactoryTest,
> TestComponentsName, TestDocBasedVersionConstraints, TestSolrCoreProperties,
> HdfsRecoveryZkTest, CoreAdminCreateDiscoverTest, TestZkChroot,
> TestSolrXmlPersistor, BJQParserTest, TestIndexSearcher,
> HdfsUnloadDistributedZkTest, TestPseudoReturnFields,
> TestCloudManagedSchemaAddField, TestPartialUpdateDeduplication,
> SuggesterFSTTest, CoreAdminHandlerTest, ChaosMonkeySafeLeaderTest,
> XmlUpdateRequestHandlerTest, TestAnalyzedSuggestions,
> LukeRequestHandlerTest, SuggesterTSTTest, CSVRequestHandlerTest,
> TestLFUCache, TestSolrQueryParserDefaultOperatorResource, StressHdfsTest,
> SpellPossibilityIteratorTest, ShardSplitTest, NotRequiredUniqueKeyTest,
> HdfsBasicDistributedZkTest, SearchHandlerTest, SimplePostToolTest,
> TestRandomDVFaceting, TestUtils, TestSystemIdResolver,
> CoreContainerCoreInitFailuresTest, TestSearchPerf, BlockDirectoryTest,
> DeleteShardTest, BasicDistributedZkTest, TestStressRecovery,
> TestNonNRTOpen, OverseerTest, SpatialFilterTest, ZkSolrClientTest,
> TestBinaryField, HighlighterTest, TestSolrDeletionPolicy1,
> TestQuerySenderListener, MigrateRouteKeyTest, URLClassifyProcessorTest,
> TestLRUCache, QueryResultKeyTest, ReturnFieldsTest, DateMathParserTest,
> JSONWriterTest, TestCSVResponseWriter, TestCustomSort,
> PathHierarchyTokenizerFactoryTest, ScriptEngineTest, RecoveryZkTest,
> ParsingFieldUpdateProcessorsTest, PolyFieldTest,
> TestSerializedLuceneMatchVersion, HdfsDirectoryTest, TestFastOutputStream,
> TimeZoneUtilsTest, TestMultiCoreConfBootstrap, QueryParsingTest,
> TestLMJelinekMercerSimilarityFactory, TestRandomMergePolicy,
> SoftAutoCommitTest, DirectSolrSpellCheckerTest, DeleteReplicaTest,
> AddBlockUpdateTest, TestIBSimilarityFactory,
> SchemaVersionSpecificBehaviorTest, SampleTest,
> TestHighFrequencyDictionaryFactory, TestSchemaNameResource,
> TestLMDirichletSimilarityFactory, ClusterStateUpdateTest,
> TestImplicitCoreProperties, TestFiltering, SortByFunctionTest,
> DebugComponentTest, TestCollationField, TestStressVersions,
> SolrTestCaseJ4Test, DocValuesMultiTest, HdfsChaosMonkeySafeLeaderTest,
> TestNumberUtils, TestDocSet, LegacyHTMLStripCharFilterTest,
> LeaderElectionTest, SolrRequestParserTest, QueryElevationComponentTest,
> DefaultValueUpdateProcessorTest, ResourceLoaderTest,
> TestQuerySenderNoQuery, TestCloudManagedSchema, ConnectionManagerTest,
> WordBreakSolrSpellCheckerTest, PingRequestHandlerTest, TestRangeQuery,
> DistanceFunctionTest, NoCacheHeaderTest, UnloadDistributedZkTest,
> CollectionsAPIDistributedZkTest]
>  [junit4] Completed in 238.84s, 1 test, 1 error <<< FAILURES!
>
> [...truncated 549 lines...]
> BUILD FAILED
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The
> following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The
> following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The
> following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The
> following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The
> following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491:
> The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308:
> The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941:
> There were test failures: 354 suites, 1540 tests, 1 error, 40 ignored (6
> assumptions)
>
> Total time: 85 minutes 3 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops
> -XX:+UseSerialGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
>


-- 
- Mark

Re: [JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1211 - Still Failing!

Posted by Mark Miller <ma...@gmail.com>.
Well, I thought it was https://issues.apache.org/jira/browse/SOLR-4805, but it’s not that, it’s just similar. 

Some clean up that still needs to be done here. It does affect 4.6.1 as well.

- Mark

On Jan 19, 2014, at 12:02 PM, Mark Miller <ma...@gmail.com> wrote:

> Well, this isn’t good. It’s an old bug brought back to life - its trying to register the core in zk again on reload - this was previously fixed. Hope it’s not in 4.6.1.
> 
> - Mark
> 
> On Jan 18, 2014, at 11:16 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:
> 
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1211/
>> Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
>> 
>> 1 tests failed.
>> FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
>> 
>> Error Message:
>> reloadcollection the collection time out:60s
>> 
>> Stack Trace:
>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: reloadcollection the collection time out:60s
>> 	at __randomizedtesting.SeedInfo.seed([A836D24B8F744B19:29D05C53F82B2B25]:0)
>> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
>> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
>> 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:765)
>> 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:196)
>> 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
>> 	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> 	at java.lang.reflect.Method.invoke(Method.java:606)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> 	at java.lang.Thread.run(Thread.java:744)
>> 
>> 
>> 
>> 
>> Build Log:
>> [...truncated 10447 lines...]
>>  [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
>>  [junit4]   2> 1753662 T5108 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
>>  [junit4]   2> 1753672 T5108 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
>>  [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1390103593043
>>  [junit4]   2> 1753674 T5108 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>>  [junit4]   2> 1753675 T5109 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>>  [junit4]   2> 1753776 T5108 oasc.ZkTestServer.run start zk server on port:54105
>>  [junit4]   2> 1753778 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1753786 T5115 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@692bd0f2 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1753787 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1753788 T5108 oascc.SolrZkClient.makePath makePath: /solr
>>  [junit4]   2> 1753800 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1753806 T5117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46284bce name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1753806 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1753807 T5108 oascc.SolrZkClient.makePath makePath: /collections/collection1
>>  [junit4]   2> 1753816 T5108 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>>  [junit4]   2> 1753824 T5108 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>>  [junit4]   2> 1753831 T5108 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>>  [junit4]   2> 1753836 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
>>  [junit4]   2> 1753837 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>>  [junit4]   2> 1753850 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
>>  [junit4]   2> 1753851 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>>  [junit4]   2> 1753861 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>>  [junit4]   2> 1753862 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>>  [junit4]   2> 1753872 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>>  [junit4]   2> 1753873 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>>  [junit4]   2> 1753886 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>>  [junit4]   2> 1753887 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>>  [junit4]   2> 1753898 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>>  [junit4]   2> 1753899 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>>  [junit4]   2> 1753909 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>>  [junit4]   2> 1753911 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>>  [junit4]   2> 1753921 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>>  [junit4]   2> 1753922 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>>  [junit4]   2> 1753930 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>>  [junit4]   2> 1753931 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>>  [junit4]   2> 1753941 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>>  [junit4]   2> 1753942 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>>  [junit4]   2> 1753953 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>>  [junit4]   2> 1753954 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>>  [junit4]   2> Using old style solr.xml
>>  [junit4]   2> 1753972 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1753978 T5119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@218b21e5 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1753978 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1753979 T5108 oascc.SolrZkClient.makePath makePath: /solr
>>  [junit4]   2> 1753986 T5110 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x143a8a1805b0002, likely client has closed socket
>>  [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
>>  [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>>  [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
>>  [junit4]   2> 
>>  [junit4]   2> 1753990 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1753999 T5121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cb1f3a name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1754000 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1754003 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
>>  [junit4]   2> 1754005 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
>>  [junit4]   2> 1754018 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
>>  [junit4]   2> 1754021 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
>>  [junit4]   2> 1754036 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>>  [junit4]   2> 1754037 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>>  [junit4]   2> 1754045 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
>>  [junit4]   2> 1754046 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
>>  [junit4]   2> 1754061 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
>>  [junit4]   2> 1754063 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
>>  [junit4]   2> 1754071 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
>>  [junit4]   2> 1754073 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
>>  [junit4]   2> 1754086 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
>>  [junit4]   2> 1754087 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
>>  [junit4]   2> 1754105 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
>>  [junit4]   2> 1754106 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
>>  [junit4]   2> 1754115 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
>>  [junit4]   2> 1754116 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
>>  [junit4]   2> 1754128 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
>>  [junit4]   2> 1754129 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
>>  [junit4]   2> 1754139 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
>>  [junit4]   2> 1754140 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
>>  [junit4]   2> 1754561 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1754575 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54110
>>  [junit4]   2> 1754576 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1754576 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1754577 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521
>>  [junit4]   2> 1754577 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/'
>>  [junit4]   2> 1754614 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solr.xml
>>  [junit4]   2> 1754697 T5108 oasc.CoreContainer.<init> New CoreContainer 725528504
>>  [junit4]   2> 1754697 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/]
>>  [junit4]   2> 1754699 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1754700 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1754700 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1754702 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1754702 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1754713 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1754714 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1754714 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1754715 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>>  [junit4]   2> 1754715 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1754716 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1754721 T5132 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20826f36 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1754722 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1754728 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1754733 T5134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ee56b7a name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1754733 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1754737 T5108 oascc.SolrZkClient.makePath makePath: /overseer/queue
>>  [junit4]   2> 1754748 T5108 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>>  [junit4]   2> 1754758 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes
>>  [junit4]   2> 1754764 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54110_
>>  [junit4]   2> 1754766 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54110_
>>  [junit4]   2> 1754776 T5108 oascc.SolrZkClient.makePath makePath: /overseer_elect
>>  [junit4]   2> 1754783 T5108 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>>  [junit4]   2> 1754796 T5108 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>>  [junit4]   2> 1754802 T5108 oasc.Overseer.start Overseer (id=91101829074190341-127.0.0.1:54110_-n_0000000000) starting
>>  [junit4]   2> 1754813 T5108 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>>  [junit4]   2> 1754826 T5136 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>>  [junit4]   2> 1754827 T5108 oascc.SolrZkClient.makePath makePath: /clusterstate.json
>>  [junit4]   2> 1754829 T5136 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
>>  [junit4]   2> 1754835 T5108 oascc.SolrZkClient.makePath makePath: /aliases.json
>>  [junit4]   2> 1754842 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1754851 T5135 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>>  [junit4]   2> 1754860 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1754861 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1754862 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1754867 T5139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bbaf1af name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1754868 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1754871 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1754879 T5108 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
>>  [junit4]   2> 1755261 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1755265 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54114
>>  [junit4]   2> 1755268 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1755268 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1755268 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250
>>  [junit4]   2> 1755269 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/'
>>  [junit4]   2> 1755308 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solr.xml
>>  [junit4]   2> 1755412 T5108 oasc.CoreContainer.<init> New CoreContainer 96924362
>>  [junit4]   2> 1755412 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/]
>>  [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1755416 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1755416 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1755429 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1755429 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1755430 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1755430 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>>  [junit4]   2> 1755430 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1755433 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1755438 T5150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@486daebc name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1755439 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1755446 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1755451 T5152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d708b92 name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1755452 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1755471 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1756488 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54114_
>>  [junit4]   2> 1756493 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54114_
>>  [junit4]   2> 1756508 T5134 oascc.ZkStateReader$3.process Updating live nodes... (2)
>>  [junit4]   2> 1756508 T5139 oascc.ZkStateReader$3.process Updating live nodes... (2)
>>  [junit4]   2> 1756510 T5152 oascc.ZkStateReader$3.process Updating live nodes... (2)
>>  [junit4]   2> 1756538 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1756539 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1756946 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1756951 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54117
>>  [junit4]   2> 1756954 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1756954 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1756954 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911
>>  [junit4]   2> 1756955 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/'
>>  [junit4]   2> 1756992 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/solr.xml
>>  [junit4]   2> 1757060 T5108 oasc.CoreContainer.<init> New CoreContainer 1734112660
>>  [junit4]   2> 1757061 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/]
>>  [junit4]   2> 1757062 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1757062 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1757074 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1757074 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1757075 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1757075 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>>  [junit4]   2> 1757075 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1757077 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1757085 T5164 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@672b7c8f name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1757086 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1757094 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1757099 T5166 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@81214c4 name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1757100 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1757110 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1758123 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54117_
>>  [junit4]   2> 1758128 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54117_
>>  [junit4]   2> 1758152 T5152 oascc.ZkStateReader$3.process Updating live nodes... (3)
>>  [junit4]   2> 1758155 T5139 oascc.ZkStateReader$3.process Updating live nodes... (3)
>>  [junit4]   2> 1758154 T5134 oascc.ZkStateReader$3.process Updating live nodes... (3)
>>  [junit4]   2> 1758155 T5166 oascc.ZkStateReader$3.process Updating live nodes... (3)
>>  [junit4]   2> 1758174 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1758175 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1758552 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1758557 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54120
>>  [junit4]   2> 1758559 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1758560 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1758560 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547
>>  [junit4]   2> 1758561 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/'
>>  [junit4]   2> 1758598 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solr.xml
>>  [junit4]   2> 1758704 T5108 oasc.CoreContainer.<init> New CoreContainer 76332827
>>  [junit4]   2> 1758705 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/]
>>  [junit4]   2> 1758706 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1758708 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1758721 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1758721 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1758722 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1758722 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>>  [junit4]   2> 1758722 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1758724 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1758728 T5178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@345687f2 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1758730 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1758737 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1758743 T5180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fde9546 name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1758744 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1758758 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1759770 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54120_
>>  [junit4]   2> 1759776 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54120_
>>  [junit4]   2> 1759791 T5166 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1759792 T5139 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1759800 T5180 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1759801 T5152 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1759802 T5134 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1759829 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1759830 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1760202 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1760208 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54123
>>  [junit4]   2> 1760210 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1760211 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1760212 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201
>>  [junit4]   2> 1760212 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/'
>>  [junit4]   2> 1760251 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solr.xml
>>  [junit4]   2> 1760338 T5108 oasc.CoreContainer.<init> New CoreContainer 723834019
>>  [junit4]   2> 1760339 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/]
>>  [junit4]   2> 1760340 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1760343 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1760343 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1760353 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1760354 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1760354 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1760355 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>>  [junit4]   2> 1760355 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1760357 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1760361 T5192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f042f75 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1760363 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1760370 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1760375 T5194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3326075a name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1760375 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1760387 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1761398 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54123_
>>  [junit4]   2> 1761407 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54123_
>>  [junit4]   2> 1761415 T5180 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1761416 T5134 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1761417 T5194 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1761417 T5152 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1761416 T5139 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1761417 T5166 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1762080 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1762080 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1762088 T5147 oasha.CollectionsHandler.handleCreateAction Creating Collection : router.field=myOwnField&version=2&name=solrj_collection&action=CREATE&wt=javabin&router.name=compositeId&numShards=2&collection.configName=conf1&replicationFactor=2
>>  [junit4]   2> 1762091 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1762094 T5136 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
>>  [junit4]   2> 	  "operation":"createcollection",
>>  [junit4]   2> 	  "fromApi":"true",
>>  [junit4]   2> 	  "name":"solrj_collection",
>>  [junit4]   2> 	  "replicationFactor":"2",
>>  [junit4]   2> 	  "collection.configName":"conf1",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "router.field":"myOwnField",
>>  [junit4]   2> 	  "router.name":"compositeId"}
>>  [junit4]   2> 1762094 T5136 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
>>  [junit4]   2> 	  "operation":"createcollection",
>>  [junit4]   2> 	  "fromApi":"true",
>>  [junit4]   2> 	  "name":"solrj_collection",
>>  [junit4]   2> 	  "replicationFactor":"2",
>>  [junit4]   2> 	  "collection.configName":"conf1",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "router.field":"myOwnField",
>>  [junit4]   2> 	  "router.name":"compositeId"}
>>  [junit4]   2> 1762095 T5136 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
>>  [junit4]   2> 1762095 T5136 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
>>  [junit4]   2> 1762107 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1762112 T5135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>>  [junit4]   2> 1762113 T5135 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
>>  [junit4]   2> 1762114 T5135 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
>>  [junit4]   2> 1762121 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1762123 T5152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762123 T5139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762124 T5194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762123 T5180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762124 T5166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762124 T5134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762208 T5136 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
>>  [junit4]   2> 1762209 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:54114_
>>  [junit4]   2> 1762209 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:54120_
>>  [junit4]   2> 1762210 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:54110_
>>  [junit4]   2> 1762210 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:54123_
>>  [junit4]   2> 1762222 T5148 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
>>  [junit4]   2> 1762223 T5126 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
>>  [junit4]   2> 1762224 T5174 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
>>  [junit4]   2> 1762224 T5190 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
>>  [junit4]   2> 1762227 T5148 oasc.ZkController.waitForCoreNodeName look for our core node name
>>  [junit4]   2> 1762228 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1762228 T5126 oasc.ZkController.waitForCoreNodeName look for our core node name
>>  [junit4]   2> 1762229 T5174 oasc.ZkController.waitForCoreNodeName look for our core node name
>>  [junit4]   2> 1762229 T5190 oasc.ZkController.waitForCoreNodeName look for our core node name
>>  [junit4]   2> 1762233 T5135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>>  [junit4]   2> 1762236 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>>  [junit4]   2> 	  "operation":"state",
>>  [junit4]   2> 	  "state":"down",
>>  [junit4]   2> 	  "base_url":"http://127.0.0.1:54114",
>>  [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
>>  [junit4]   2> 	  "roles":null,
>>  [junit4]   2> 	  "node_name":"127.0.0.1:54114_",
>>  [junit4]   2> 	  "shard":"shard1",
>>  [junit4]   2> 	  "shard_range":null,
>>  [junit4]   2> 	  "shard_state":"active",
>>  [junit4]   2> 	  "shard_parent":null,
>>  [junit4]   2> 	  "collection":"solrj_collection",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "core_node_name":null}
>>  [junit4]   2> 1762243 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1762246 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>>  [junit4]   2> 	  "operation":"state",
>>  [junit4]   2> 	  "state":"down",
>>  [junit4]   2> 	  "base_url":"http://127.0.0.1:54110",
>>  [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
>>  [junit4]   2> 	  "roles":null,
>>  [junit4]   2> 	  "node_name":"127.0.0.1:54110_",
>>  [junit4]   2> 	  "shard":"shard2",
>>  [junit4]   2> 	  "shard_range":null,
>>  [junit4]   2> 	  "shard_state":"active",
>>  [junit4]   2> 	  "shard_parent":null,
>>  [junit4]   2> 	  "collection":"solrj_collection",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "core_node_name":null}
>>  [junit4]   2> 1762254 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1762257 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>>  [junit4]   2> 	  "operation":"state",
>>  [junit4]   2> 	  "state":"down",
>>  [junit4]   2> 	  "base_url":"http://127.0.0.1:54123",
>>  [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
>>  [junit4]   2> 	  "roles":null,
>>  [junit4]   2> 	  "node_name":"127.0.0.1:54123_",
>>  [junit4]   2> 	  "shard":"shard2",
>>  [junit4]   2> 	  "shard_range":null,
>>  [junit4]   2> 	  "shard_state":"active",
>>  [junit4]   2> 	  "shard_parent":null,
>>  [junit4]   2> 	  "collection":"solrj_collection",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "core_node_name":null}
>>  [junit4]   2> 1762265 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1762267 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>>  [junit4]   2> 	  "operation":"state",
>>  [junit4]   2> 	  "state":"down",
>>  [junit4]   2> 	  "base_url":"http://127.0.0.1:54120",
>>  [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
>>  [junit4]   2> 	  "roles":null,
>>  [junit4]   2> 	  "node_name":"127.0.0.1:54120_",
>>  [junit4]   2> 	  "shard":"shard1",
>>  [junit4]   2> 	  "shard_range":null,
>>  [junit4]   2> 	  "shard_state":"active",
>>  [junit4]   2> 	  "shard_parent":null,
>>  [junit4]   2> 	  "collection":"solrj_collection",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "core_node_name":null}
>>  [junit4]   2> 1762276 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1762383 T5152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762383 T5166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762383 T5134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762383 T5139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762383 T5194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1762383 T5180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1763229 T5148 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
>>  [junit4]   2> 1763230 T5148 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solr.xml
>>  [junit4]   2> 1763231 T5148 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1
>>  [junit4]   2> 1763231 T5148 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>>  [junit4]   2> 1763230 T5126 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
>>  [junit4]   2> 1763233 T5148 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>>  [junit4]   2> 1763233 T5126 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solr.xml
>>  [junit4]   2> 1763234 T5126 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1
>>  [junit4]   2> 1763234 T5126 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>>  [junit4]   2> 1763231 T5190 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
>>  [junit4]   2> 1763231 T5174 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
>>  [junit4]   2> 1763236 T5126 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>>  [junit4]   2> 1763237 T5126 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>>  [junit4]   2> 1763236 T5190 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solr.xml
>>  [junit4]   2> 1763238 T5190 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2
>>  [junit4]   2> 1763239 T5190 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>>  [junit4]   2> 1763233 T5148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>>  [junit4]   2> 1763240 T5126 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/'
>>  [junit4]   2> 1763237 T5174 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solr.xml
>>  [junit4]   2> 1763243 T5174 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2
>>  [junit4]   2> 1763243 T5174 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>>  [junit4]   2> 1763242 T5190 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>>  [junit4]   2> 1763245 T5190 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>>  [junit4]   2> 1763247 T5174 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>>  [junit4]   2> 1763250 T5174 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>>  [junit4]   2> 1763249 T5148 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/'
>>  [junit4]   2> 1763269 T5190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/'
>>  [junit4]   2> 1763277 T5174 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/'
>>  [junit4]   2> 1763420 T5148 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>>  [junit4]   2> 1763477 T5126 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>>  [junit4]   2> 1763482 T5190 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>>  [junit4]   2> 1763477 T5174 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>>  [junit4]   2> 1763684 T5174 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>>  [junit4]   2> 1763688 T5174 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>>  [junit4]   2> 1763689 T5126 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>>  [junit4]   2> 1763703 T5174 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
>>  [junit4]   2> 1763718 T5126 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>>  [junit4]   2> 1763735 T5126 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
>>  [junit4]   2> 1763767 T5148 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>>  [junit4]   2> 1763798 T5190 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>>  [junit4]   2> 1763809 T5148 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>>  [junit4]   2> 1763813 T5190 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>>  [junit4]   2> 1763838 T5190 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
>>  [junit4]   2> 1763843 T5148 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
>>  [junit4]   2> 1765449 T5126 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1765457 T5126 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1765458 T5126 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>>  [junit4]   2> 1765534 T5126 oass.IndexSchema.readSchema default search field in schema is text
>>  [junit4]   2> 1765545 T5126 oass.IndexSchema.readSchema unique key field: id
>>  [junit4]   2> 1765568 T5148 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1765568 T5148 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1765599 T5174 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1765604 T5174 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1765605 T5174 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>>  [junit4]   2> 1765612 T5174 oass.IndexSchema.readSchema default search field in schema is text
>>  [junit4]   2> 1765601 T5148 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>>  [junit4]   2> 1765635 T5174 oass.IndexSchema.readSchema unique key field: id
>>  [junit4]   2> 1765649 T5148 oass.IndexSchema.readSchema default search field in schema is text
>>  [junit4]   2> 1765665 T5148 oass.IndexSchema.readSchema unique key field: id
>>  [junit4]   2> 1765675 T5126 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1765685 T5126 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1765693 T5126 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1765698 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1765699 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1765699 T5126 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1765704 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1765705 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1765705 T5126 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>>  [junit4]   2> 1765706 T5126 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/
>>  [junit4]   2> 1765707 T5126 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>>  [junit4]   2> 1765706 T5190 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1765712 T5174 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1765713 T5190 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1765711 T5126 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data
>>  [junit4]   2> 1765721 T5174 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1765718 T5190 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>>  [junit4]   2> 1765730 T5190 oass.IndexSchema.readSchema default search field in schema is text
>>  [junit4]   2> 1765735 T5174 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1765722 T5126 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index/
>>  [junit4]   2> 1765739 T5126 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
>>  [junit4]   2> 1765739 T5126 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index
>>  [junit4]   2> 1765740 T5126 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>>  [junit4]   2> 1765740 T5190 oass.IndexSchema.readSchema unique key field: id
>>  [junit4]   2> 1765743 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1765746 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1765749 T5174 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1765745 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1765756 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1765757 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1765758 T5174 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>>  [junit4]   2> 1765758 T5174 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/
>>  [junit4]   2> 1765758 T5174 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>>  [junit4]   2> 1765748 T5126 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7477e0e2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e4480e6),segFN=segments_1,generation=1}
>>  [junit4]   2> 1765760 T5126 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1765761 T5174 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data
>>  [junit4]   2> 1765768 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>>  [junit4]   2> 1765768 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1765769 T5174 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index/
>>  [junit4]   2> 1765770 T5174 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
>>  [junit4]   2> 1765771 T5174 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index
>>  [junit4]   2> 1765773 T5174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>>  [junit4]   2> 1765775 T5174 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4a94a9c3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cd26ede),segFN=segments_1,generation=1}
>>  [junit4]   2> 1765775 T5174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1765778 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1765769 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1765780 T5126 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1765780 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>>  [junit4]   2> 1765781 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1765780 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1765783 T5126 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1765784 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>>  [junit4]   2> 1765782 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1765785 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1765786 T5174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1765786 T5174 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>>  [junit4]   2> 1765788 T5174 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>>  [junit4]   2> 1765789 T5174 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>>  [junit4]   2> 1765789 T5174 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>>  [junit4]   2> 1765791 T5174 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>>  [junit4]   2> 1765785 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1765792 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1765784 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>>  [junit4]   2> 1765792 T5174 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>>  [junit4]   2> 1765795 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1765795 T5174 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>>  [junit4]   2> 1765797 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1765799 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1765799 T5148 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>>  [junit4]   2> 1765797 T5126 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1765800 T5126 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>>  [junit4]   2> 1765800 T5148 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/
>>  [junit4]   2> 1765802 T5126 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>>  [junit4]   2> 1765802 T5190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1765798 T5174 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>>  [junit4]   2> 1765803 T5126 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>>  [junit4]   2> 1765803 T5126 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>>  [junit4]   2> 1765804 T5126 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>>  [junit4]   2> 1765802 T5148 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>>  [junit4]   2> 1765807 T5126 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>>  [junit4]   2> 1765809 T5126 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>>  [junit4]   2> 1765810 T5126 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>>  [junit4]   2> 1765813 T5148 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data
>>  [junit4]   2> 1765814 T5148 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index/
>>  [junit4]   2> 1765814 T5148 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
>>  [junit4]   2> 1765816 T5126 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>>  [junit4]   2> 1765820 T5148 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index
>>  [junit4]   2> 1765820 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>>  [junit4]   2> 1765817 T5174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>>  [junit4]   2> 1765823 T5190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1765823 T5126 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>>  [junit4]   2> 1765825 T5126 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>>  [junit4]   2> 1765826 T5126 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=892100269, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
>>  [junit4]   2> 1765827 T5126 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7477e0e2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e4480e6),segFN=segments_1,generation=1}
>>  [junit4]   2> 1765827 T5126 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1765828 T5126 oass.SolrIndexSearcher.<init> Opening Searcher@190a2642 main
>>  [junit4]   2> 1765829 T5148 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@775d1479 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75a7b414),segFN=segments_1,generation=1}
>>  [junit4]   2> 1765830 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1765834 T5200 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica1] Registered new searcher Searcher@190a2642 main{StandardDirectoryReader(segments_1:1:nrt)}
>>  [junit4]   2> 1765837 T5126 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica1
>>  [junit4]   2> 1765837 T5126 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica1 address:http://127.0.0.1:54110 collection:solrj_collection shard:shard2
>>  [junit4]   2> 1765841 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>>  [junit4]   2> 1765841 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1765842 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>>  [junit4]   2> 1765844 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1765844 T5126 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
>>  [junit4]   2> 1765841 T5190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1765845 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1765846 T5148 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>>  [junit4]   2> 1765842 T5174 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>>  [junit4]   2> 1765848 T5174 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>>  [junit4]   2> 1765847 T5148 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>>  [junit4]   2> 1765851 T5174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=892100269, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
>>  [junit4]   2> 1765854 T5174 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4a94a9c3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cd26ede),segFN=segments_1,generation=1}
>>  [junit4]   2> 1765854 T5174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1765855 T5174 oass.SolrIndexSearcher.<init> Opening Searcher@290c4ce6 main
>>  [junit4]   2> 1765852 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1765857 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1765858 T5190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1765858 T5148 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>>  [junit4]   2> 1765859 T5148 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>>  [junit4]   2> 1765859 T5148 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>>  [junit4]   2> 1765860 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>>  [junit4]   2> 1765860 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>>  [junit4]   2> 1765861 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>>  [junit4]   2> 1765865 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>>  [junit4]   2> 1765865 T5201 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@290c4ce6 main{StandardDirectoryReader(segments_1:1:nrt)}
>>  [junit4]   2> 1765869 T5174 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
>>  [junit4]   2> 1765870 T5174 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:http://127.0.0.1:54120 collection:solrj_collection shard:shard1
>>  [junit4]   2> 1765872 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1765873 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1765873 T5190 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>>  [junit4]   2> 1765874 T5190 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/
>>  [junit4]   2> 1765875 T5190 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>>  [junit4]   2> 1765876 T5174 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
>>  [junit4]   2> 1765876 T5190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data
>>  [junit4]   2> 1765878 T5190 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index/
>>  [junit4]   2> 1765878 T5190 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
>>  [junit4]   2> 1765878 T5190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index
>>  [junit4]   2> 1765879 T5190 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>>  [junit4]   2> 1765879 T5148 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>>  [junit4]   2> 1765880 T5148 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>>  [junit4]   2> 1765881 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=892100269, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
>>  [junit4]   2> 1765883 T5148 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@775d1479 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75a7b414),segFN=segments_1,generation=1}
>>  [junit4]   2> 1765885 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1765886 T5148 oass.SolrIndexSearcher.<init> Opening Searcher@3dd43c40 main
>>  [junit4]   2> 1765884 T5190 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@40bfb0df lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6da86f2c),segFN=segments_1,generation=1}
>>  [junit4]   2> 1765889 T5190 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1765893 T5204 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@3dd43c40 main{StandardDirectoryReader(segments_1:1:nrt)}
>>  [junit4]   2> 1765896 T5148 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
>>  [junit4]   2> 1765897 T5148 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:http://127.0.0.1:54114 collection:solrj_collection shard:shard1
>>  [junit4]   2> 1765896 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>>  [junit4]   2> 1765899 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1765896 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
>>  [junit4]   2> 1765901 T5190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1765903 T5148 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
>>  [junit4]   2> 1765909 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1765911 T5190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1765912 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>>  [junit4]   2> 1765913 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1765913 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1765918 T5190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1765919 T5190 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>>  [junit4]   2> 1765918 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1765920 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1765921 T5190 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>>  [junit4]   2> 1765922 T5190 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>>  [junit4]   2> 1765923 T5190 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>>  [junit4]   2> 1765924 T5190 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>>  [junit4]   2> 1765925 T5126 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=-8
>>  [junit4]   2> 1765926 T5126 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
>>  [junit4]   2> 1765927 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>>  [junit4]   2> ASYNC  NEW_CORE C3406 name=solrj_collection_shard2_replica1 org.apache.solr.core.SolrCore@5da01731 url=http://127.0.0.1:54110/solrj_collection_shard2_replica1 node=127.0.0.1:54110_ C3406_STATE=coll:solrj_collection core:solrj_collection_shard2_replica1 props:{state=down, base_url=http://127.0.0.1:54110, core=solrj_collection_shard2_replica1, node_name=127.0.0.1:54110_}
>>  [junit4]   2> 1765929 T5126 C3406 P54110 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54110/solrj_collection_shard2_replica1/
>>  [junit4]   2> 1765932 T5126 C3406 P54110 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard2_replica1 url=http://127.0.0.1:54110 START replicas=[http://127.0.0.1:54123/solrj_collection_shard2_replica2/] nUpdates=100
>>  [junit4]   2> 1765933 T5126 C3406 P54110 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard2_replica1 url=http://127.0.0.1:54110 DONE.  We have no versions.  sync failed.
>>  [junit4]   2> 1765934 T5126 C3406 P54110 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
>>  [junit4]   2> 1765935 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
>>  [junit4]   2> 1765936 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54110/solrj_collection_shard2_replica1/ shard2
>>  [junit4]   2> 1765938 T5126 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard2
>>  [junit4]   2> 1765926 T5190 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>>  [junit4]   2> 1765932 T5135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>>  [junit4]   2> 
>> 
>> [...truncated too long message...]
>> 
>> ZkCmdExecutor.java:73)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>>  [junit4]   2> 
>>  [junit4]   2> 1991623 T5194 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>>  [junit4]   2> 1991623 T5194 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_2/leader_elect/shard1/election
>>  [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>>  [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>  [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>>  [junit4]   2> 
>>  [junit4]   2> 1991623 T5194 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_0/leader_elect/shard3/election
>>  [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>>  [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>  [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>>  [junit4]   2> 
>>  [junit4]   2> 1991804 T5180 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>>  [junit4]   2> NOTE: test params are: codec=Lucene46: {range_facet_si=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), range_facet_sl=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=Memory doPackFST= false), a_si=Lucene41(blocksize=128), other_tl1=Lucene41(blocksize=128), text=PostingsFormat(name=MockRandom), multiDefault=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), intDefault=Lucene41(blocksize=128), timestamp=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), a_t=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), range_facet_l=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), _version_=Lucene41(blocksize=128)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=de_LU, timezone=CET
>>  [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=3,free=99625584,total=338677760
>>  [junit4]   2> NOTE: All tests run in this JVM: [TestSolrJ, TestDistributedGrouping, TestFreeTextSuggestions, TestSchemaSimilarityResource, DirectUpdateHandlerTest, PreAnalyzedUpdateProcessorTest, TestCoreDiscovery, CoreMergeIndexesAdminHandlerTest, TestClassNameShortening, IndexSchemaTest, TestWordDelimiterFilterFactory, FullSolrCloudDistribCmdsTest, TestShardHandlerFactory, CircularListTest, TestRealTimeGet, SolrPluginUtilsTest, TestQueryUtils, TestExtendedDismaxParser, TestSchemaVersionResource, TestPostingsSolrHighlighter, UniqFieldsUpdateProcessorFactoryTest, MoreLikeThisHandlerTest, TestPhraseSuggestions, TestFuzzyAnalyzedSuggestions, UpdateParamsTest, TestInfoStreamLogging, BasicDistributedZk2Test, SyncSliceTest, TestPerFieldSimilarity, TestNRTOpen, BasicFunctionalityTest, TestAtomicUpdateErrorCases, SliceStateUpdateTest, JsonLoaderTest, TestMaxScoreQueryParser, BadIndexSchemaTest, PrimUtilsTest, TestMergePolicyConfig, BadCopyFieldTest, TestOmitPositions, TestRandomFaceting, RAMDirectoryFactoryTest, RequestHandlersTest, AliasIntegrationTest, HighlighterConfigTest, TestFileDictionaryLookup, TestLuceneMatchVersion, TestDefaultSearchFieldResource, TriLevelCompositeIdRoutingTest, UUIDFieldTest, TestSurroundQueryParser, StandardRequestHandlerTest, TestHashPartitioner, RegexBoostProcessorTest, TestBM25SimilarityFactory, SuggesterWFSTTest, DocumentBuilderTest, TestSolrDeletionPolicy2, SuggesterTest, OutputWriterTest, SimpleFacetsTest, FieldAnalysisRequestHandlerTest, TestFieldTypeResource, SolrIndexSplitterTest, TestPHPSerializedResponseWriter, SpellCheckComponentTest, UpdateRequestProcessorFactoryTest, TestComponentsName, TestDocBasedVersionConstraints, TestSolrCoreProperties, HdfsRecoveryZkTest, CoreAdminCreateDiscoverTest, TestZkChroot, TestSolrXmlPersistor, BJQParserTest, TestIndexSearcher, HdfsUnloadDistributedZkTest, TestPseudoReturnFields, TestCloudManagedSchemaAddField, TestPartialUpdateDeduplication, SuggesterFSTTest, CoreAdminHandlerTest, ChaosMonkeySafeLeaderTest, XmlUpdateRequestHandlerTest, TestAnalyzedSuggestions, LukeRequestHandlerTest, SuggesterTSTTest, CSVRequestHandlerTest, TestLFUCache, TestSolrQueryParserDefaultOperatorResource, StressHdfsTest, SpellPossibilityIteratorTest, ShardSplitTest, NotRequiredUniqueKeyTest, HdfsBasicDistributedZkTest, SearchHandlerTest, SimplePostToolTest, TestRandomDVFaceting, TestUtils, TestSystemIdResolver, CoreContainerCoreInitFailuresTest, TestSearchPerf, BlockDirectoryTest, DeleteShardTest, BasicDistributedZkTest, TestStressRecovery, TestNonNRTOpen, OverseerTest, SpatialFilterTest, ZkSolrClientTest, TestBinaryField, HighlighterTest, TestSolrDeletionPolicy1, TestQuerySenderListener, MigrateRouteKeyTest, URLClassifyProcessorTest, TestLRUCache, QueryResultKeyTest, ReturnFieldsTest, DateMathParserTest, JSONWriterTest, TestCSVResponseWriter, TestCustomSort, PathHierarchyTokenizerFactoryTest, ScriptEngineTest, RecoveryZkTest, ParsingFieldUpdateProcessorsTest, PolyFieldTest, TestSerializedLuceneMatchVersion, HdfsDirectoryTest, TestFastOutputStream, TimeZoneUtilsTest, TestMultiCoreConfBootstrap, QueryParsingTest, TestLMJelinekMercerSimilarityFactory, TestRandomMergePolicy, SoftAutoCommitTest, DirectSolrSpellCheckerTest, DeleteReplicaTest, AddBlockUpdateTest, TestIBSimilarityFactory, SchemaVersionSpecificBehaviorTest, SampleTest, TestHighFrequencyDictionaryFactory, TestSchemaNameResource, TestLMDirichletSimilarityFactory, ClusterStateUpdateTest, TestImplicitCoreProperties, TestFiltering, SortByFunctionTest, DebugComponentTest, TestCollationField, TestStressVersions, SolrTestCaseJ4Test, DocValuesMultiTest, HdfsChaosMonkeySafeLeaderTest, TestNumberUtils, TestDocSet, LegacyHTMLStripCharFilterTest, LeaderElectionTest, SolrRequestParserTest, QueryElevationComponentTest, DefaultValueUpdateProcessorTest, ResourceLoaderTest, TestQuerySenderNoQuery, TestCloudManagedSchema, ConnectionManagerTest, WordBreakSolrSpellCheckerTest, PingRequestHandlerTest, TestRangeQuery, DistanceFunctionTest, NoCacheHeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
>>  [junit4] Completed in 238.84s, 1 test, 1 error <<< FAILURES!
>> 
>> [...truncated 549 lines...]
>> BUILD FAILED
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1540 tests, 1 error, 40 ignored (6 assumptions)
>> 
>> Total time: 85 minutes 3 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
> 


Re: [JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1211 - Still Failing!

Posted by Mark Miller <ma...@gmail.com>.
Well, this isn’t good. It’s an old bug brought back to life - its trying to register the core in zk again on reload - this was previously fixed. Hope it’s not in 4.6.1.

- Mark

On Jan 18, 2014, at 11:16 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1211/
> Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
> 
> 1 tests failed.
> FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
> 
> Error Message:
> reloadcollection the collection time out:60s
> 
> Stack Trace:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: reloadcollection the collection time out:60s
> 	at __randomizedtesting.SeedInfo.seed([A836D24B8F744B19:29D05C53F82B2B25]:0)
> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
> 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:765)
> 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:196)
> 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
> 	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> 	at java.lang.Thread.run(Thread.java:744)
> 
> 
> 
> 
> Build Log:
> [...truncated 10447 lines...]
>   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
>   [junit4]   2> 1753662 T5108 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
>   [junit4]   2> 1753672 T5108 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
>   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1390103593043
>   [junit4]   2> 1753674 T5108 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>   [junit4]   2> 1753675 T5109 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>   [junit4]   2> 1753776 T5108 oasc.ZkTestServer.run start zk server on port:54105
>   [junit4]   2> 1753778 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1753786 T5115 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@692bd0f2 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1753787 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1753788 T5108 oascc.SolrZkClient.makePath makePath: /solr
>   [junit4]   2> 1753800 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1753806 T5117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46284bce name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1753806 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1753807 T5108 oascc.SolrZkClient.makePath makePath: /collections/collection1
>   [junit4]   2> 1753816 T5108 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>   [junit4]   2> 1753824 T5108 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>   [junit4]   2> 1753831 T5108 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>   [junit4]   2> 1753836 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
>   [junit4]   2> 1753837 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>   [junit4]   2> 1753850 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
>   [junit4]   2> 1753851 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>   [junit4]   2> 1753861 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 1753862 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 1753872 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>   [junit4]   2> 1753873 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>   [junit4]   2> 1753886 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>   [junit4]   2> 1753887 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>   [junit4]   2> 1753898 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>   [junit4]   2> 1753899 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>   [junit4]   2> 1753909 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>   [junit4]   2> 1753911 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>   [junit4]   2> 1753921 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 1753922 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 1753930 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 1753931 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 1753941 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>   [junit4]   2> 1753942 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>   [junit4]   2> 1753953 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>   [junit4]   2> 1753954 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>   [junit4]   2> Using old style solr.xml
>   [junit4]   2> 1753972 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1753978 T5119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@218b21e5 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1753978 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1753979 T5108 oascc.SolrZkClient.makePath makePath: /solr
>   [junit4]   2> 1753986 T5110 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x143a8a1805b0002, likely client has closed socket
>   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
>   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
>   [junit4]   2> 
>   [junit4]   2> 1753990 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1753999 T5121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cb1f3a name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1754000 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1754003 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
>   [junit4]   2> 1754005 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
>   [junit4]   2> 1754018 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
>   [junit4]   2> 1754021 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
>   [junit4]   2> 1754036 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 1754037 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 1754045 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
>   [junit4]   2> 1754046 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
>   [junit4]   2> 1754061 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
>   [junit4]   2> 1754063 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
>   [junit4]   2> 1754071 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
>   [junit4]   2> 1754073 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
>   [junit4]   2> 1754086 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
>   [junit4]   2> 1754087 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
>   [junit4]   2> 1754105 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 1754106 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 1754115 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
>   [junit4]   2> 1754116 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
>   [junit4]   2> 1754128 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
>   [junit4]   2> 1754129 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
>   [junit4]   2> 1754139 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
>   [junit4]   2> 1754140 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
>   [junit4]   2> 1754561 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1754575 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54110
>   [junit4]   2> 1754576 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1754576 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1754577 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521
>   [junit4]   2> 1754577 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/'
>   [junit4]   2> 1754614 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solr.xml
>   [junit4]   2> 1754697 T5108 oasc.CoreContainer.<init> New CoreContainer 725528504
>   [junit4]   2> 1754697 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/]
>   [junit4]   2> 1754699 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1754700 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1754700 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1754702 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1754702 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1754713 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1754714 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1754714 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1754715 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>   [junit4]   2> 1754715 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1754716 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1754721 T5132 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20826f36 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1754722 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1754728 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1754733 T5134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ee56b7a name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1754733 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1754737 T5108 oascc.SolrZkClient.makePath makePath: /overseer/queue
>   [junit4]   2> 1754748 T5108 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>   [junit4]   2> 1754758 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes
>   [junit4]   2> 1754764 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54110_
>   [junit4]   2> 1754766 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54110_
>   [junit4]   2> 1754776 T5108 oascc.SolrZkClient.makePath makePath: /overseer_elect
>   [junit4]   2> 1754783 T5108 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>   [junit4]   2> 1754796 T5108 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>   [junit4]   2> 1754802 T5108 oasc.Overseer.start Overseer (id=91101829074190341-127.0.0.1:54110_-n_0000000000) starting
>   [junit4]   2> 1754813 T5108 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>   [junit4]   2> 1754826 T5136 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>   [junit4]   2> 1754827 T5108 oascc.SolrZkClient.makePath makePath: /clusterstate.json
>   [junit4]   2> 1754829 T5136 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
>   [junit4]   2> 1754835 T5108 oascc.SolrZkClient.makePath makePath: /aliases.json
>   [junit4]   2> 1754842 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1754851 T5135 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>   [junit4]   2> 1754860 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1754861 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1754862 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1754867 T5139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bbaf1af name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1754868 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1754871 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1754879 T5108 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
>   [junit4]   2> 1755261 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1755265 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54114
>   [junit4]   2> 1755268 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1755268 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1755268 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250
>   [junit4]   2> 1755269 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/'
>   [junit4]   2> 1755308 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solr.xml
>   [junit4]   2> 1755412 T5108 oasc.CoreContainer.<init> New CoreContainer 96924362
>   [junit4]   2> 1755412 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/]
>   [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1755416 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1755416 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1755429 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1755429 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1755430 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1755430 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>   [junit4]   2> 1755430 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1755433 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1755438 T5150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@486daebc name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1755439 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1755446 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1755451 T5152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d708b92 name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1755452 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1755471 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1756488 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54114_
>   [junit4]   2> 1756493 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54114_
>   [junit4]   2> 1756508 T5134 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 1756508 T5139 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 1756510 T5152 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 1756538 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1756539 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1756946 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1756951 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54117
>   [junit4]   2> 1756954 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1756954 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1756954 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911
>   [junit4]   2> 1756955 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/'
>   [junit4]   2> 1756992 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/solr.xml
>   [junit4]   2> 1757060 T5108 oasc.CoreContainer.<init> New CoreContainer 1734112660
>   [junit4]   2> 1757061 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/]
>   [junit4]   2> 1757062 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1757062 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1757074 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1757074 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1757075 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1757075 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>   [junit4]   2> 1757075 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1757077 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1757085 T5164 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@672b7c8f name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1757086 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1757094 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1757099 T5166 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@81214c4 name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1757100 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1757110 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1758123 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54117_
>   [junit4]   2> 1758128 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54117_
>   [junit4]   2> 1758152 T5152 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 1758155 T5139 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 1758154 T5134 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 1758155 T5166 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 1758174 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1758175 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1758552 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1758557 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54120
>   [junit4]   2> 1758559 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1758560 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1758560 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547
>   [junit4]   2> 1758561 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/'
>   [junit4]   2> 1758598 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solr.xml
>   [junit4]   2> 1758704 T5108 oasc.CoreContainer.<init> New CoreContainer 76332827
>   [junit4]   2> 1758705 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/]
>   [junit4]   2> 1758706 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1758708 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1758721 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1758721 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1758722 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1758722 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>   [junit4]   2> 1758722 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1758724 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1758728 T5178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@345687f2 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1758730 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1758737 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1758743 T5180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fde9546 name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1758744 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1758758 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1759770 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54120_
>   [junit4]   2> 1759776 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54120_
>   [junit4]   2> 1759791 T5166 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1759792 T5139 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1759800 T5180 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1759801 T5152 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1759802 T5134 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1759829 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1759830 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1760202 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1760208 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54123
>   [junit4]   2> 1760210 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1760211 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1760212 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201
>   [junit4]   2> 1760212 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/'
>   [junit4]   2> 1760251 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solr.xml
>   [junit4]   2> 1760338 T5108 oasc.CoreContainer.<init> New CoreContainer 723834019
>   [junit4]   2> 1760339 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/]
>   [junit4]   2> 1760340 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1760343 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1760343 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1760353 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1760354 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1760354 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1760355 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
>   [junit4]   2> 1760355 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1760357 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1760361 T5192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f042f75 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1760363 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1760370 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1760375 T5194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3326075a name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1760375 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1760387 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1761398 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54123_
>   [junit4]   2> 1761407 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54123_
>   [junit4]   2> 1761415 T5180 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1761416 T5134 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1761417 T5194 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1761417 T5152 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1761416 T5139 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1761417 T5166 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1762080 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1762080 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1762088 T5147 oasha.CollectionsHandler.handleCreateAction Creating Collection : router.field=myOwnField&version=2&name=solrj_collection&action=CREATE&wt=javabin&router.name=compositeId&numShards=2&collection.configName=conf1&replicationFactor=2
>   [junit4]   2> 1762091 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1762094 T5136 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
>   [junit4]   2> 	  "operation":"createcollection",
>   [junit4]   2> 	  "fromApi":"true",
>   [junit4]   2> 	  "name":"solrj_collection",
>   [junit4]   2> 	  "replicationFactor":"2",
>   [junit4]   2> 	  "collection.configName":"conf1",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "router.field":"myOwnField",
>   [junit4]   2> 	  "router.name":"compositeId"}
>   [junit4]   2> 1762094 T5136 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
>   [junit4]   2> 	  "operation":"createcollection",
>   [junit4]   2> 	  "fromApi":"true",
>   [junit4]   2> 	  "name":"solrj_collection",
>   [junit4]   2> 	  "replicationFactor":"2",
>   [junit4]   2> 	  "collection.configName":"conf1",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "router.field":"myOwnField",
>   [junit4]   2> 	  "router.name":"compositeId"}
>   [junit4]   2> 1762095 T5136 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
>   [junit4]   2> 1762095 T5136 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
>   [junit4]   2> 1762107 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1762112 T5135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 1762113 T5135 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
>   [junit4]   2> 1762114 T5135 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
>   [junit4]   2> 1762121 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1762123 T5152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762123 T5139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762124 T5194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762123 T5180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762124 T5166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762124 T5134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762208 T5136 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
>   [junit4]   2> 1762209 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:54114_
>   [junit4]   2> 1762209 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:54120_
>   [junit4]   2> 1762210 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:54110_
>   [junit4]   2> 1762210 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:54123_
>   [junit4]   2> 1762222 T5148 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
>   [junit4]   2> 1762223 T5126 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
>   [junit4]   2> 1762224 T5174 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
>   [junit4]   2> 1762224 T5190 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
>   [junit4]   2> 1762227 T5148 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 1762228 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1762228 T5126 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 1762229 T5174 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 1762229 T5190 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 1762233 T5135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 1762236 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:54114",
>   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:54114_",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"solrj_collection",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 1762243 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1762246 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:54110",
>   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:54110_",
>   [junit4]   2> 	  "shard":"shard2",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"solrj_collection",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 1762254 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1762257 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:54123",
>   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:54123_",
>   [junit4]   2> 	  "shard":"shard2",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"solrj_collection",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 1762265 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1762267 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:54120",
>   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:54120_",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"solrj_collection",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 1762276 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1762383 T5152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762383 T5166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762383 T5134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762383 T5139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762383 T5194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1762383 T5180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1763229 T5148 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
>   [junit4]   2> 1763230 T5148 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solr.xml
>   [junit4]   2> 1763231 T5148 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1
>   [junit4]   2> 1763231 T5148 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>   [junit4]   2> 1763230 T5126 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
>   [junit4]   2> 1763233 T5148 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 1763233 T5126 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solr.xml
>   [junit4]   2> 1763234 T5126 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1
>   [junit4]   2> 1763234 T5126 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>   [junit4]   2> 1763231 T5190 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
>   [junit4]   2> 1763231 T5174 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
>   [junit4]   2> 1763236 T5126 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 1763237 T5126 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>   [junit4]   2> 1763236 T5190 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solr.xml
>   [junit4]   2> 1763238 T5190 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2
>   [junit4]   2> 1763239 T5190 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>   [junit4]   2> 1763233 T5148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>   [junit4]   2> 1763240 T5126 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/'
>   [junit4]   2> 1763237 T5174 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solr.xml
>   [junit4]   2> 1763243 T5174 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2
>   [junit4]   2> 1763243 T5174 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>   [junit4]   2> 1763242 T5190 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 1763245 T5190 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>   [junit4]   2> 1763247 T5174 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 1763250 T5174 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>   [junit4]   2> 1763249 T5148 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/'
>   [junit4]   2> 1763269 T5190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/'
>   [junit4]   2> 1763277 T5174 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/'
>   [junit4]   2> 1763420 T5148 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 1763477 T5126 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 1763482 T5190 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 1763477 T5174 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 1763684 T5174 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 1763688 T5174 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 1763689 T5126 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 1763703 T5174 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
>   [junit4]   2> 1763718 T5126 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 1763735 T5126 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
>   [junit4]   2> 1763767 T5148 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 1763798 T5190 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 1763809 T5148 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 1763813 T5190 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 1763838 T5190 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
>   [junit4]   2> 1763843 T5148 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
>   [junit4]   2> 1765449 T5126 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1765457 T5126 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1765458 T5126 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 1765534 T5126 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 1765545 T5126 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 1765568 T5148 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1765568 T5148 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1765599 T5174 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1765604 T5174 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1765605 T5174 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 1765612 T5174 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 1765601 T5148 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 1765635 T5174 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 1765649 T5148 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 1765665 T5148 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 1765675 T5126 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1765685 T5126 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1765693 T5126 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1765698 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1765699 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1765699 T5126 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1765704 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1765705 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1765705 T5126 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 1765706 T5126 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/
>   [junit4]   2> 1765707 T5126 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>   [junit4]   2> 1765706 T5190 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1765712 T5174 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1765713 T5190 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1765711 T5126 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data
>   [junit4]   2> 1765721 T5174 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1765718 T5190 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 1765730 T5190 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 1765735 T5174 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1765722 T5126 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index/
>   [junit4]   2> 1765739 T5126 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 1765739 T5126 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index
>   [junit4]   2> 1765740 T5126 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>   [junit4]   2> 1765740 T5190 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 1765743 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1765746 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1765749 T5174 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1765745 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1765756 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1765757 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1765758 T5174 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 1765758 T5174 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/
>   [junit4]   2> 1765758 T5174 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>   [junit4]   2> 1765748 T5126 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7477e0e2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e4480e6),segFN=segments_1,generation=1}
>   [junit4]   2> 1765760 T5126 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1765761 T5174 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data
>   [junit4]   2> 1765768 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 1765768 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1765769 T5174 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index/
>   [junit4]   2> 1765770 T5174 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 1765771 T5174 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index
>   [junit4]   2> 1765773 T5174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>   [junit4]   2> 1765775 T5174 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4a94a9c3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cd26ede),segFN=segments_1,generation=1}
>   [junit4]   2> 1765775 T5174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1765778 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1765769 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1765780 T5126 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1765780 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 1765781 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1765780 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1765783 T5126 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1765784 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 1765782 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1765785 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1765786 T5174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1765786 T5174 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 1765788 T5174 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 1765789 T5174 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 1765789 T5174 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 1765791 T5174 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 1765785 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1765792 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1765784 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 1765792 T5174 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 1765795 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1765795 T5174 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 1765797 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1765799 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1765799 T5148 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 1765797 T5126 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1765800 T5126 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 1765800 T5148 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/
>   [junit4]   2> 1765802 T5126 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 1765802 T5190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1765798 T5174 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 1765803 T5126 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 1765803 T5126 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 1765804 T5126 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 1765802 T5148 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>   [junit4]   2> 1765807 T5126 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 1765809 T5126 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 1765810 T5126 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 1765813 T5148 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data
>   [junit4]   2> 1765814 T5148 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index/
>   [junit4]   2> 1765814 T5148 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 1765816 T5126 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 1765820 T5148 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index
>   [junit4]   2> 1765820 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>   [junit4]   2> 1765817 T5174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 1765823 T5190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1765823 T5126 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 1765825 T5126 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 1765826 T5126 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=892100269, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
>   [junit4]   2> 1765827 T5126 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7477e0e2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e4480e6),segFN=segments_1,generation=1}
>   [junit4]   2> 1765827 T5126 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1765828 T5126 oass.SolrIndexSearcher.<init> Opening Searcher@190a2642 main
>   [junit4]   2> 1765829 T5148 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@775d1479 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75a7b414),segFN=segments_1,generation=1}
>   [junit4]   2> 1765830 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1765834 T5200 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica1] Registered new searcher Searcher@190a2642 main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 1765837 T5126 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica1
>   [junit4]   2> 1765837 T5126 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica1 address:http://127.0.0.1:54110 collection:solrj_collection shard:shard2
>   [junit4]   2> 1765841 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 1765841 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1765842 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 1765844 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1765844 T5126 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
>   [junit4]   2> 1765841 T5190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1765845 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1765846 T5148 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 1765842 T5174 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 1765848 T5174 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 1765847 T5148 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 1765851 T5174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=892100269, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
>   [junit4]   2> 1765854 T5174 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4a94a9c3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cd26ede),segFN=segments_1,generation=1}
>   [junit4]   2> 1765854 T5174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1765855 T5174 oass.SolrIndexSearcher.<init> Opening Searcher@290c4ce6 main
>   [junit4]   2> 1765852 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1765857 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1765858 T5190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1765858 T5148 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 1765859 T5148 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 1765859 T5148 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 1765860 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 1765860 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 1765861 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 1765865 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 1765865 T5201 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@290c4ce6 main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 1765869 T5174 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
>   [junit4]   2> 1765870 T5174 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:http://127.0.0.1:54120 collection:solrj_collection shard:shard1
>   [junit4]   2> 1765872 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1765873 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1765873 T5190 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 1765874 T5190 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/
>   [junit4]   2> 1765875 T5190 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
>   [junit4]   2> 1765876 T5174 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
>   [junit4]   2> 1765876 T5190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data
>   [junit4]   2> 1765878 T5190 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index/
>   [junit4]   2> 1765878 T5190 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 1765878 T5190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index
>   [junit4]   2> 1765879 T5190 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
>   [junit4]   2> 1765879 T5148 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 1765880 T5148 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 1765881 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=892100269, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
>   [junit4]   2> 1765883 T5148 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@775d1479 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75a7b414),segFN=segments_1,generation=1}
>   [junit4]   2> 1765885 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1765886 T5148 oass.SolrIndexSearcher.<init> Opening Searcher@3dd43c40 main
>   [junit4]   2> 1765884 T5190 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@40bfb0df lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6da86f2c),segFN=segments_1,generation=1}
>   [junit4]   2> 1765889 T5190 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1765893 T5204 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@3dd43c40 main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 1765896 T5148 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
>   [junit4]   2> 1765897 T5148 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:http://127.0.0.1:54114 collection:solrj_collection shard:shard1
>   [junit4]   2> 1765896 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 1765899 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1765896 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
>   [junit4]   2> 1765901 T5190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1765903 T5148 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
>   [junit4]   2> 1765909 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1765911 T5190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1765912 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 1765913 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1765913 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1765918 T5190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1765919 T5190 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 1765918 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1765920 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1765921 T5190 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 1765922 T5190 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 1765923 T5190 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 1765924 T5190 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 1765925 T5126 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=-8
>   [junit4]   2> 1765926 T5126 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
>   [junit4]   2> 1765927 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> ASYNC  NEW_CORE C3406 name=solrj_collection_shard2_replica1 org.apache.solr.core.SolrCore@5da01731 url=http://127.0.0.1:54110/solrj_collection_shard2_replica1 node=127.0.0.1:54110_ C3406_STATE=coll:solrj_collection core:solrj_collection_shard2_replica1 props:{state=down, base_url=http://127.0.0.1:54110, core=solrj_collection_shard2_replica1, node_name=127.0.0.1:54110_}
>   [junit4]   2> 1765929 T5126 C3406 P54110 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54110/solrj_collection_shard2_replica1/
>   [junit4]   2> 1765932 T5126 C3406 P54110 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard2_replica1 url=http://127.0.0.1:54110 START replicas=[http://127.0.0.1:54123/solrj_collection_shard2_replica2/] nUpdates=100
>   [junit4]   2> 1765933 T5126 C3406 P54110 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard2_replica1 url=http://127.0.0.1:54110 DONE.  We have no versions.  sync failed.
>   [junit4]   2> 1765934 T5126 C3406 P54110 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
>   [junit4]   2> 1765935 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
>   [junit4]   2> 1765936 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54110/solrj_collection_shard2_replica1/ shard2
>   [junit4]   2> 1765938 T5126 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard2
>   [junit4]   2> 1765926 T5190 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 1765932 T5135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 
> 
> [...truncated too long message...]
> 
> ZkCmdExecutor.java:73)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 
>   [junit4]   2> 1991623 T5194 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>   [junit4]   2> 1991623 T5194 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_2/leader_elect/shard1/election
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 
>   [junit4]   2> 1991623 T5194 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_0/leader_elect/shard3/election
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 
>   [junit4]   2> 1991804 T5180 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>   [junit4]   2> NOTE: test params are: codec=Lucene46: {range_facet_si=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), range_facet_sl=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=Memory doPackFST= false), a_si=Lucene41(blocksize=128), other_tl1=Lucene41(blocksize=128), text=PostingsFormat(name=MockRandom), multiDefault=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), intDefault=Lucene41(blocksize=128), timestamp=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), a_t=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), range_facet_l=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), _version_=Lucene41(blocksize=128)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=de_LU, timezone=CET
>   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=3,free=99625584,total=338677760
>   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrJ, TestDistributedGrouping, TestFreeTextSuggestions, TestSchemaSimilarityResource, DirectUpdateHandlerTest, PreAnalyzedUpdateProcessorTest, TestCoreDiscovery, CoreMergeIndexesAdminHandlerTest, TestClassNameShortening, IndexSchemaTest, TestWordDelimiterFilterFactory, FullSolrCloudDistribCmdsTest, TestShardHandlerFactory, CircularListTest, TestRealTimeGet, SolrPluginUtilsTest, TestQueryUtils, TestExtendedDismaxParser, TestSchemaVersionResource, TestPostingsSolrHighlighter, UniqFieldsUpdateProcessorFactoryTest, MoreLikeThisHandlerTest, TestPhraseSuggestions, TestFuzzyAnalyzedSuggestions, UpdateParamsTest, TestInfoStreamLogging, BasicDistributedZk2Test, SyncSliceTest, TestPerFieldSimilarity, TestNRTOpen, BasicFunctionalityTest, TestAtomicUpdateErrorCases, SliceStateUpdateTest, JsonLoaderTest, TestMaxScoreQueryParser, BadIndexSchemaTest, PrimUtilsTest, TestMergePolicyConfig, BadCopyFieldTest, TestOmitPositions, TestRandomFaceting, RAMDirectoryFactoryTest, RequestHandlersTest, AliasIntegrationTest, HighlighterConfigTest, TestFileDictionaryLookup, TestLuceneMatchVersion, TestDefaultSearchFieldResource, TriLevelCompositeIdRoutingTest, UUIDFieldTest, TestSurroundQueryParser, StandardRequestHandlerTest, TestHashPartitioner, RegexBoostProcessorTest, TestBM25SimilarityFactory, SuggesterWFSTTest, DocumentBuilderTest, TestSolrDeletionPolicy2, SuggesterTest, OutputWriterTest, SimpleFacetsTest, FieldAnalysisRequestHandlerTest, TestFieldTypeResource, SolrIndexSplitterTest, TestPHPSerializedResponseWriter, SpellCheckComponentTest, UpdateRequestProcessorFactoryTest, TestComponentsName, TestDocBasedVersionConstraints, TestSolrCoreProperties, HdfsRecoveryZkTest, CoreAdminCreateDiscoverTest, TestZkChroot, TestSolrXmlPersistor, BJQParserTest, TestIndexSearcher, HdfsUnloadDistributedZkTest, TestPseudoReturnFields, TestCloudManagedSchemaAddField, TestPartialUpdateDeduplication, SuggesterFSTTest, CoreAdminHandlerTest, ChaosMonkeySafeLeaderTest, XmlUpdateRequestHandlerTest, TestAnalyzedSuggestions, LukeRequestHandlerTest, SuggesterTSTTest, CSVRequestHandlerTest, TestLFUCache, TestSolrQueryParserDefaultOperatorResource, StressHdfsTest, SpellPossibilityIteratorTest, ShardSplitTest, NotRequiredUniqueKeyTest, HdfsBasicDistributedZkTest, SearchHandlerTest, SimplePostToolTest, TestRandomDVFaceting, TestUtils, TestSystemIdResolver, CoreContainerCoreInitFailuresTest, TestSearchPerf, BlockDirectoryTest, DeleteShardTest, BasicDistributedZkTest, TestStressRecovery, TestNonNRTOpen, OverseerTest, SpatialFilterTest, ZkSolrClientTest, TestBinaryField, HighlighterTest, TestSolrDeletionPolicy1, TestQuerySenderListener, MigrateRouteKeyTest, URLClassifyProcessorTest, TestLRUCache, QueryResultKeyTest, ReturnFieldsTest, DateMathParserTest, JSONWriterTest, TestCSVResponseWriter, TestCustomSort, PathHierarchyTokenizerFactoryTest, ScriptEngineTest, RecoveryZkTest, ParsingFieldUpdateProcessorsTest, PolyFieldTest, TestSerializedLuceneMatchVersion, HdfsDirectoryTest, TestFastOutputStream, TimeZoneUtilsTest, TestMultiCoreConfBootstrap, QueryParsingTest, TestLMJelinekMercerSimilarityFactory, TestRandomMergePolicy, SoftAutoCommitTest, DirectSolrSpellCheckerTest, DeleteReplicaTest, AddBlockUpdateTest, TestIBSimilarityFactory, SchemaVersionSpecificBehaviorTest, SampleTest, TestHighFrequencyDictionaryFactory, TestSchemaNameResource, TestLMDirichletSimilarityFactory, ClusterStateUpdateTest, TestImplicitCoreProperties, TestFiltering, SortByFunctionTest, DebugComponentTest, TestCollationField, TestStressVersions, SolrTestCaseJ4Test, DocValuesMultiTest, HdfsChaosMonkeySafeLeaderTest, TestNumberUtils, TestDocSet, LegacyHTMLStripCharFilterTest, LeaderElectionTest, SolrRequestParserTest, QueryElevationComponentTest, DefaultValueUpdateProcessorTest, ResourceLoaderTest, TestQuerySenderNoQuery, TestCloudManagedSchema, ConnectionManagerTest, WordBreakSolrSpellCheckerTest, PingRequestHandlerTest, TestRangeQuery, DistanceFunctionTest, NoCacheHeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
>   [junit4] Completed in 238.84s, 1 test, 1 error <<< FAILURES!
> 
> [...truncated 549 lines...]
> BUILD FAILED
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1540 tests, 1 error, 40 ignored (6 assumptions)
> 
> Total time: 85 minutes 3 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1211 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1211/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
reloadcollection the collection time out:60s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: reloadcollection the collection time out:60s
	at __randomizedtesting.SeedInfo.seed([A836D24B8F744B19:29D05C53F82B2B25]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:765)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:196)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10447 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 1753662 T5108 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1753672 T5108 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1390103593043
   [junit4]   2> 1753674 T5108 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1753675 T5109 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1753776 T5108 oasc.ZkTestServer.run start zk server on port:54105
   [junit4]   2> 1753778 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1753786 T5115 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@692bd0f2 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1753787 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1753788 T5108 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1753800 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1753806 T5117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46284bce name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1753806 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1753807 T5108 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1753816 T5108 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1753824 T5108 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1753831 T5108 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1753836 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1753837 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1753850 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1753851 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1753861 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1753862 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1753872 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1753873 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1753886 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1753887 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1753898 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1753899 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1753909 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1753911 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1753921 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1753922 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1753930 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1753931 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1753941 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1753942 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1753953 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1753954 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 1753972 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1753978 T5119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@218b21e5 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1753978 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1753979 T5108 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1753986 T5110 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x143a8a1805b0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 1753990 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1753999 T5121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cb1f3a name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1754000 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1754003 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 1754005 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 1754018 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 1754021 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 1754036 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1754037 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1754045 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 1754046 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 1754061 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 1754063 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 1754071 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 1754073 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 1754086 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1754087 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1754105 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1754106 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1754115 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 1754116 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 1754128 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 1754129 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 1754139 T5108 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 1754140 T5108 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 1754561 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1754575 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54110
   [junit4]   2> 1754576 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1754576 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1754577 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521
   [junit4]   2> 1754577 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/'
   [junit4]   2> 1754614 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solr.xml
   [junit4]   2> 1754697 T5108 oasc.CoreContainer.<init> New CoreContainer 725528504
   [junit4]   2> 1754697 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/]
   [junit4]   2> 1754699 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1754700 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1754700 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1754701 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1754702 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1754702 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1754713 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1754714 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1754714 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1754715 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
   [junit4]   2> 1754715 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1754716 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1754721 T5132 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20826f36 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1754722 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1754728 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1754733 T5134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ee56b7a name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1754733 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1754737 T5108 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1754748 T5108 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1754758 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1754764 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54110_
   [junit4]   2> 1754766 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54110_
   [junit4]   2> 1754776 T5108 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1754783 T5108 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1754796 T5108 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1754802 T5108 oasc.Overseer.start Overseer (id=91101829074190341-127.0.0.1:54110_-n_0000000000) starting
   [junit4]   2> 1754813 T5108 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1754826 T5136 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1754827 T5108 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1754829 T5136 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1754835 T5108 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1754842 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1754851 T5135 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1754860 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1754861 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1754862 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1754867 T5139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bbaf1af name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1754868 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1754871 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1754879 T5108 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1755261 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1755265 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54114
   [junit4]   2> 1755268 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1755268 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1755268 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250
   [junit4]   2> 1755269 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/'
   [junit4]   2> 1755308 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solr.xml
   [junit4]   2> 1755412 T5108 oasc.CoreContainer.<init> New CoreContainer 96924362
   [junit4]   2> 1755412 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/]
   [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1755415 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1755416 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1755416 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1755417 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1755429 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1755429 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1755430 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1755430 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
   [junit4]   2> 1755430 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1755433 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1755438 T5150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@486daebc name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1755439 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1755446 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1755451 T5152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d708b92 name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1755452 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1755471 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1756488 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54114_
   [junit4]   2> 1756493 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54114_
   [junit4]   2> 1756508 T5134 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1756508 T5139 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1756510 T5152 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1756538 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1756539 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1756946 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1756951 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54117
   [junit4]   2> 1756954 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1756954 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1756954 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911
   [junit4]   2> 1756955 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/'
   [junit4]   2> 1756992 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/solr.xml
   [junit4]   2> 1757060 T5108 oasc.CoreContainer.<init> New CoreContainer 1734112660
   [junit4]   2> 1757061 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390103595911/]
   [junit4]   2> 1757062 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1757062 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1757063 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1757064 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1757074 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1757074 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1757075 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1757075 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
   [junit4]   2> 1757075 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1757077 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1757085 T5164 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@672b7c8f name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1757086 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1757094 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1757099 T5166 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@81214c4 name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1757100 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1757110 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1758123 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54117_
   [junit4]   2> 1758128 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54117_
   [junit4]   2> 1758152 T5152 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1758155 T5139 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1758154 T5134 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1758155 T5166 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1758174 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1758175 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1758552 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1758557 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54120
   [junit4]   2> 1758559 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1758560 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1758560 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547
   [junit4]   2> 1758561 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/'
   [junit4]   2> 1758598 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solr.xml
   [junit4]   2> 1758704 T5108 oasc.CoreContainer.<init> New CoreContainer 76332827
   [junit4]   2> 1758705 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/]
   [junit4]   2> 1758706 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1758708 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1758709 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1758710 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1758721 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1758721 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1758722 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1758722 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
   [junit4]   2> 1758722 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1758724 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1758728 T5178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@345687f2 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1758730 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1758737 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1758743 T5180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fde9546 name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1758744 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1758758 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1759770 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54120_
   [junit4]   2> 1759776 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54120_
   [junit4]   2> 1759791 T5166 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1759792 T5139 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1759800 T5180 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1759801 T5152 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1759802 T5134 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1759829 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1759830 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1760202 T5108 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1760208 T5108 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54123
   [junit4]   2> 1760210 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1760211 T5108 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1760212 T5108 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201
   [junit4]   2> 1760212 T5108 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/'
   [junit4]   2> 1760251 T5108 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solr.xml
   [junit4]   2> 1760338 T5108 oasc.CoreContainer.<init> New CoreContainer 723834019
   [junit4]   2> 1760339 T5108 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/]
   [junit4]   2> 1760340 T5108 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1760341 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1760342 T5108 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1760343 T5108 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1760343 T5108 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1760353 T5108 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1760354 T5108 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1760354 T5108 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1760355 T5108 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54105/solr
   [junit4]   2> 1760355 T5108 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1760357 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1760361 T5192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f042f75 name:ZooKeeperConnection Watcher:127.0.0.1:54105 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1760363 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1760370 T5108 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1760375 T5194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3326075a name:ZooKeeperConnection Watcher:127.0.0.1:54105/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1760375 T5108 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1760387 T5108 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1761398 T5108 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54123_
   [junit4]   2> 1761407 T5108 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54123_
   [junit4]   2> 1761415 T5180 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1761416 T5134 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1761417 T5194 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1761417 T5152 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1761416 T5139 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1761417 T5166 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1762080 T5108 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1762080 T5108 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1762088 T5147 oasha.CollectionsHandler.handleCreateAction Creating Collection : router.field=myOwnField&version=2&name=solrj_collection&action=CREATE&wt=javabin&router.name=compositeId&numShards=2&collection.configName=conf1&replicationFactor=2
   [junit4]   2> 1762091 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1762094 T5136 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 1762094 T5136 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 1762095 T5136 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1762095 T5136 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 1762107 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1762112 T5135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1762113 T5135 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 1762114 T5135 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 1762121 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1762123 T5152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762123 T5139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762124 T5194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762123 T5180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762124 T5166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762124 T5134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762208 T5136 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 1762209 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:54114_
   [junit4]   2> 1762209 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:54120_
   [junit4]   2> 1762210 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:54110_
   [junit4]   2> 1762210 T5136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:54123_
   [junit4]   2> 1762222 T5148 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 1762223 T5126 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 1762224 T5174 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 1762224 T5190 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 1762227 T5148 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1762228 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1762228 T5126 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1762229 T5174 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1762229 T5190 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1762233 T5135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1762236 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54114",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54114_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1762243 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1762246 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54110",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54110_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1762254 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1762257 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54123",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54123_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1762265 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1762267 T5135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54120",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54120_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1762276 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1762383 T5152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762383 T5166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762383 T5134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762383 T5139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762383 T5194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1762383 T5180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1763229 T5148 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1763230 T5148 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solr.xml
   [junit4]   2> 1763231 T5148 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1
   [junit4]   2> 1763231 T5148 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1763230 T5126 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1763233 T5148 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1763233 T5126 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solr.xml
   [junit4]   2> 1763234 T5126 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1
   [junit4]   2> 1763234 T5126 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1763231 T5190 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1763231 T5174 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1763236 T5126 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1763237 T5126 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1763236 T5190 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solr.xml
   [junit4]   2> 1763238 T5190 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2
   [junit4]   2> 1763239 T5190 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1763233 T5148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1763240 T5126 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/'
   [junit4]   2> 1763237 T5174 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solr.xml
   [junit4]   2> 1763243 T5174 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2
   [junit4]   2> 1763243 T5174 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1763242 T5190 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1763245 T5190 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1763247 T5174 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1763250 T5174 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1763249 T5148 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/'
   [junit4]   2> 1763269 T5190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/'
   [junit4]   2> 1763277 T5174 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/'
   [junit4]   2> 1763420 T5148 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1763477 T5126 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1763482 T5190 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1763477 T5174 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1763684 T5174 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1763688 T5174 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1763689 T5126 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1763703 T5174 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 1763718 T5126 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1763735 T5126 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 1763767 T5148 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1763798 T5190 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1763809 T5148 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1763813 T5190 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1763838 T5190 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 1763843 T5148 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 1765449 T5126 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1765457 T5126 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1765458 T5126 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1765534 T5126 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1765545 T5126 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1765568 T5148 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1765568 T5148 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1765599 T5174 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1765604 T5174 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1765605 T5174 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1765612 T5174 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1765601 T5148 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1765635 T5174 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1765649 T5148 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1765665 T5148 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1765675 T5126 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1765685 T5126 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1765693 T5126 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1765698 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1765699 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1765699 T5126 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1765704 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1765705 T5126 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1765705 T5126 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1765706 T5126 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/
   [junit4]   2> 1765707 T5126 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
   [junit4]   2> 1765706 T5190 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1765712 T5174 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1765713 T5190 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1765711 T5126 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data
   [junit4]   2> 1765721 T5174 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1765718 T5190 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1765730 T5190 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1765735 T5174 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1765722 T5126 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 1765739 T5126 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1765739 T5126 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390103593521/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 1765740 T5126 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
   [junit4]   2> 1765740 T5190 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1765743 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1765746 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1765749 T5174 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1765745 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1765756 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1765757 T5174 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1765758 T5174 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1765758 T5174 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/
   [junit4]   2> 1765758 T5174 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
   [junit4]   2> 1765748 T5126 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7477e0e2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e4480e6),segFN=segments_1,generation=1}
   [junit4]   2> 1765760 T5126 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1765761 T5174 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data
   [junit4]   2> 1765768 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1765768 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1765769 T5174 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 1765770 T5174 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1765771 T5174 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390103597547/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 1765773 T5174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
   [junit4]   2> 1765775 T5174 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4a94a9c3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cd26ede),segFN=segments_1,generation=1}
   [junit4]   2> 1765775 T5174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1765778 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1765769 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1765780 T5126 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1765780 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1765781 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1765780 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1765783 T5126 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1765782 T5174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1765784 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1765782 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1765785 T5174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1765786 T5174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1765786 T5174 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1765788 T5174 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1765789 T5174 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1765789 T5174 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1765791 T5174 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1765785 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1765792 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1765784 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1765792 T5174 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1765795 T5126 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1765795 T5174 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1765797 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1765799 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1765799 T5148 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1765797 T5126 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1765800 T5126 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1765800 T5148 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/
   [junit4]   2> 1765802 T5126 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1765802 T5190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1765798 T5174 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1765803 T5126 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1765803 T5126 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1765804 T5126 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1765802 T5148 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
   [junit4]   2> 1765807 T5126 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1765809 T5126 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1765810 T5126 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1765813 T5148 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data
   [junit4]   2> 1765814 T5148 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 1765814 T5148 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1765816 T5126 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1765820 T5148 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390103594250/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 1765820 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
   [junit4]   2> 1765817 T5174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1765823 T5190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1765823 T5126 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1765825 T5126 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1765826 T5126 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=892100269, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1765827 T5126 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7477e0e2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e4480e6),segFN=segments_1,generation=1}
   [junit4]   2> 1765827 T5126 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1765828 T5126 oass.SolrIndexSearcher.<init> Opening Searcher@190a2642 main
   [junit4]   2> 1765829 T5148 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@775d1479 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75a7b414),segFN=segments_1,generation=1}
   [junit4]   2> 1765830 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1765834 T5200 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica1] Registered new searcher Searcher@190a2642 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1765837 T5126 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica1
   [junit4]   2> 1765837 T5126 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica1 address:http://127.0.0.1:54110 collection:solrj_collection shard:shard2
   [junit4]   2> 1765841 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1765841 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1765842 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1765843 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1765844 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1765844 T5126 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 1765841 T5190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1765845 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1765846 T5148 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1765842 T5174 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1765848 T5174 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1765847 T5148 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1765851 T5174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=892100269, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1765854 T5174 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4a94a9c3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cd26ede),segFN=segments_1,generation=1}
   [junit4]   2> 1765854 T5174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1765855 T5174 oass.SolrIndexSearcher.<init> Opening Searcher@290c4ce6 main
   [junit4]   2> 1765852 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1765857 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1765858 T5190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1765858 T5148 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1765859 T5148 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1765859 T5148 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1765860 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1765860 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1765861 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1765865 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1765865 T5201 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@290c4ce6 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1765869 T5174 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 1765870 T5174 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:http://127.0.0.1:54120 collection:solrj_collection shard:shard1
   [junit4]   2> 1765872 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1765873 T5190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1765873 T5190 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1765874 T5190 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/
   [junit4]   2> 1765875 T5190 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48580d0c
   [junit4]   2> 1765876 T5174 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 1765876 T5190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data
   [junit4]   2> 1765878 T5190 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 1765878 T5190 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1765878 T5190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390103599201/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 1765879 T5190 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6052431603391322]
   [junit4]   2> 1765879 T5148 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1765880 T5148 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1765881 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=892100269, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1765883 T5148 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@775d1479 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75a7b414),segFN=segments_1,generation=1}
   [junit4]   2> 1765885 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1765886 T5148 oass.SolrIndexSearcher.<init> Opening Searcher@3dd43c40 main
   [junit4]   2> 1765884 T5190 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@40bfb0df lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6da86f2c),segFN=segments_1,generation=1}
   [junit4]   2> 1765889 T5190 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1765893 T5204 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@3dd43c40 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1765896 T5148 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
   [junit4]   2> 1765897 T5148 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:http://127.0.0.1:54114 collection:solrj_collection shard:shard1
   [junit4]   2> 1765896 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1765899 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1765896 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1765901 T5190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1765903 T5148 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 1765909 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1765911 T5190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1765912 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1765913 T5190 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1765913 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1765918 T5190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1765919 T5190 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1765918 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1765920 T5134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1765921 T5190 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1765922 T5190 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1765923 T5190 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1765924 T5190 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1765925 T5126 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=-8
   [junit4]   2> 1765926 T5126 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
   [junit4]   2> 1765927 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3406 name=solrj_collection_shard2_replica1 org.apache.solr.core.SolrCore@5da01731 url=http://127.0.0.1:54110/solrj_collection_shard2_replica1 node=127.0.0.1:54110_ C3406_STATE=coll:solrj_collection core:solrj_collection_shard2_replica1 props:{state=down, base_url=http://127.0.0.1:54110, core=solrj_collection_shard2_replica1, node_name=127.0.0.1:54110_}
   [junit4]   2> 1765929 T5126 C3406 P54110 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54110/solrj_collection_shard2_replica1/
   [junit4]   2> 1765932 T5126 C3406 P54110 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard2_replica1 url=http://127.0.0.1:54110 START replicas=[http://127.0.0.1:54123/solrj_collection_shard2_replica2/] nUpdates=100
   [junit4]   2> 1765933 T5126 C3406 P54110 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard2_replica1 url=http://127.0.0.1:54110 DONE.  We have no versions.  sync failed.
   [junit4]   2> 1765934 T5126 C3406 P54110 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1765935 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1765936 T5126 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54110/solrj_collection_shard2_replica1/ shard2
   [junit4]   2> 1765938 T5126 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard2
   [junit4]   2> 1765926 T5190 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1765932 T5135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 

[...truncated too long message...]

ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1991623 T5194 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1991623 T5194 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_2/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1991623 T5194 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_0/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1991804 T5180 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {range_facet_si=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), range_facet_sl=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=Memory doPackFST= false), a_si=Lucene41(blocksize=128), other_tl1=Lucene41(blocksize=128), text=PostingsFormat(name=MockRandom), multiDefault=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), intDefault=Lucene41(blocksize=128), timestamp=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), a_t=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), range_facet_l=Pulsing41(freqCutoff=1 minBlockSize=29 maxBlockSize=136), _version_=Lucene41(blocksize=128)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=de_LU, timezone=CET
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=3,free=99625584,total=338677760
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrJ, TestDistributedGrouping, TestFreeTextSuggestions, TestSchemaSimilarityResource, DirectUpdateHandlerTest, PreAnalyzedUpdateProcessorTest, TestCoreDiscovery, CoreMergeIndexesAdminHandlerTest, TestClassNameShortening, IndexSchemaTest, TestWordDelimiterFilterFactory, FullSolrCloudDistribCmdsTest, TestShardHandlerFactory, CircularListTest, TestRealTimeGet, SolrPluginUtilsTest, TestQueryUtils, TestExtendedDismaxParser, TestSchemaVersionResource, TestPostingsSolrHighlighter, UniqFieldsUpdateProcessorFactoryTest, MoreLikeThisHandlerTest, TestPhraseSuggestions, TestFuzzyAnalyzedSuggestions, UpdateParamsTest, TestInfoStreamLogging, BasicDistributedZk2Test, SyncSliceTest, TestPerFieldSimilarity, TestNRTOpen, BasicFunctionalityTest, TestAtomicUpdateErrorCases, SliceStateUpdateTest, JsonLoaderTest, TestMaxScoreQueryParser, BadIndexSchemaTest, PrimUtilsTest, TestMergePolicyConfig, BadCopyFieldTest, TestOmitPositions, TestRandomFaceting, RAMDirectoryFactoryTest, RequestHandlersTest, AliasIntegrationTest, HighlighterConfigTest, TestFileDictionaryLookup, TestLuceneMatchVersion, TestDefaultSearchFieldResource, TriLevelCompositeIdRoutingTest, UUIDFieldTest, TestSurroundQueryParser, StandardRequestHandlerTest, TestHashPartitioner, RegexBoostProcessorTest, TestBM25SimilarityFactory, SuggesterWFSTTest, DocumentBuilderTest, TestSolrDeletionPolicy2, SuggesterTest, OutputWriterTest, SimpleFacetsTest, FieldAnalysisRequestHandlerTest, TestFieldTypeResource, SolrIndexSplitterTest, TestPHPSerializedResponseWriter, SpellCheckComponentTest, UpdateRequestProcessorFactoryTest, TestComponentsName, TestDocBasedVersionConstraints, TestSolrCoreProperties, HdfsRecoveryZkTest, CoreAdminCreateDiscoverTest, TestZkChroot, TestSolrXmlPersistor, BJQParserTest, TestIndexSearcher, HdfsUnloadDistributedZkTest, TestPseudoReturnFields, TestCloudManagedSchemaAddField, TestPartialUpdateDeduplication, SuggesterFSTTest, CoreAdminHandlerTest, ChaosMonkeySafeLeaderTest, XmlUpdateRequestHandlerTest, TestAnalyzedSuggestions, LukeRequestHandlerTest, SuggesterTSTTest, CSVRequestHandlerTest, TestLFUCache, TestSolrQueryParserDefaultOperatorResource, StressHdfsTest, SpellPossibilityIteratorTest, ShardSplitTest, NotRequiredUniqueKeyTest, HdfsBasicDistributedZkTest, SearchHandlerTest, SimplePostToolTest, TestRandomDVFaceting, TestUtils, TestSystemIdResolver, CoreContainerCoreInitFailuresTest, TestSearchPerf, BlockDirectoryTest, DeleteShardTest, BasicDistributedZkTest, TestStressRecovery, TestNonNRTOpen, OverseerTest, SpatialFilterTest, ZkSolrClientTest, TestBinaryField, HighlighterTest, TestSolrDeletionPolicy1, TestQuerySenderListener, MigrateRouteKeyTest, URLClassifyProcessorTest, TestLRUCache, QueryResultKeyTest, ReturnFieldsTest, DateMathParserTest, JSONWriterTest, TestCSVResponseWriter, TestCustomSort, PathHierarchyTokenizerFactoryTest, ScriptEngineTest, RecoveryZkTest, ParsingFieldUpdateProcessorsTest, PolyFieldTest, TestSerializedLuceneMatchVersion, HdfsDirectoryTest, TestFastOutputStream, TimeZoneUtilsTest, TestMultiCoreConfBootstrap, QueryParsingTest, TestLMJelinekMercerSimilarityFactory, TestRandomMergePolicy, SoftAutoCommitTest, DirectSolrSpellCheckerTest, DeleteReplicaTest, AddBlockUpdateTest, TestIBSimilarityFactory, SchemaVersionSpecificBehaviorTest, SampleTest, TestHighFrequencyDictionaryFactory, TestSchemaNameResource, TestLMDirichletSimilarityFactory, ClusterStateUpdateTest, TestImplicitCoreProperties, TestFiltering, SortByFunctionTest, DebugComponentTest, TestCollationField, TestStressVersions, SolrTestCaseJ4Test, DocValuesMultiTest, HdfsChaosMonkeySafeLeaderTest, TestNumberUtils, TestDocSet, LegacyHTMLStripCharFilterTest, LeaderElectionTest, SolrRequestParserTest, QueryElevationComponentTest, DefaultValueUpdateProcessorTest, ResourceLoaderTest, TestQuerySenderNoQuery, TestCloudManagedSchema, ConnectionManagerTest, WordBreakSolrSpellCheckerTest, PingRequestHandlerTest, TestRangeQuery, DistanceFunctionTest, NoCacheHeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed in 238.84s, 1 test, 1 error <<< FAILURES!

[...truncated 549 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1540 tests, 1 error, 40 ignored (6 assumptions)

Total time: 85 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1210 - Still Failing!

Posted by Mark Miller <ma...@gmail.com>.
Hmm…I must not have committed that change. I know I had done it locally recently, so figured it was causing this - it’s relatively new I think.

So, perhaps the 2 minute so timeout is long enough that a 330 second wait might not be enough…

I’ll try raising the wait in the test and see if it still trips.

- Mark

On Jan 18, 2014, at 6:35 PM, Mark Miller <ma...@gmail.com> wrote:

> Looks like it’s stuck in a socket read. I’ll turn the tests distributed so timeout back on.
> 
> - Mark
> 
> On Jan 18, 2014, at 4:22 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:
> 
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1210/
>> Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
>> 
>> 1 tests failed.
>> FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
>> 
>> Error Message:
>> There are still nodes recoverying - waited for 330 seconds
>> 
>> Stack Trace:
>> java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
>> 	at __randomizedtesting.SeedInfo.seed([E01973DE033FE27B:61FFFDC674608247]:0)
>> 	at org.junit.Assert.fail(Assert.java:93)
>> 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
>> 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
>> 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
>> 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:772)
>> 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:196)
>> 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
>> 	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> 	at java.lang.reflect.Method.invoke(Method.java:606)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> 	at java.lang.Thread.run(Thread.java:744)
>> 
>> 
>> 
>> 
>> Build Log:
>> [...truncated 10429 lines...]
>>  [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
>>  [junit4]   2> 1719213 T5915 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /od_jjq/cq
>>  [junit4]   2> 1719222 T5915 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
>>  [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1390078610952
>>  [junit4]   2> 1719224 T5915 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>>  [junit4]   2> 1719225 T5916 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>>  [junit4]   2> 1719327 T5915 oasc.ZkTestServer.run start zk server on port:53689
>>  [junit4]   2> 1719329 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1719338 T5922 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b2bf8c1 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1719338 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1719339 T5915 oascc.SolrZkClient.makePath makePath: /solr
>>  [junit4]   2> 1719353 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1719358 T5924 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31e8fc2a name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1719359 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1719359 T5915 oascc.SolrZkClient.makePath makePath: /collections/collection1
>>  [junit4]   2> 1719375 T5915 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>>  [junit4]   2> 1719384 T5915 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>>  [junit4]   2> 1719393 T5915 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>>  [junit4]   2> 1719403 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
>>  [junit4]   2> 1719404 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>>  [junit4]   2> 1719421 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
>>  [junit4]   2> 1719422 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>>  [junit4]   2> 1719432 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>>  [junit4]   2> 1719433 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>>  [junit4]   2> 1719444 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>>  [junit4]   2> 1719445 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>>  [junit4]   2> 1719456 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>>  [junit4]   2> 1719457 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>>  [junit4]   2> 1719469 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>>  [junit4]   2> 1719470 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>>  [junit4]   2> 1719480 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>>  [junit4]   2> 1719481 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>>  [junit4]   2> 1719492 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>>  [junit4]   2> 1719493 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>>  [junit4]   2> 1719503 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>>  [junit4]   2> 1719504 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>>  [junit4]   2> 1719513 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>>  [junit4]   2> 1719514 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>>  [junit4]   2> 1719524 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>>  [junit4]   2> 1719526 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>>  [junit4]   2> Using old style solr.xml
>>  [junit4]   2> 1719542 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1719547 T5926 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66930f41 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1719547 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1719548 T5915 oascc.SolrZkClient.makePath makePath: /solr
>>  [junit4]   2> 1719558 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1719562 T5928 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3239c4d9 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1719562 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1719564 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
>>  [junit4]   2> 1719565 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
>>  [junit4]   2> 1719576 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
>>  [junit4]   2> 1719578 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
>>  [junit4]   2> 1719588 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>>  [junit4]   2> 1719589 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>>  [junit4]   2> 1719600 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
>>  [junit4]   2> 1719600 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
>>  [junit4]   2> 1719611 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
>>  [junit4]   2> 1719612 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
>>  [junit4]   2> 1719622 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
>>  [junit4]   2> 1719624 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
>>  [junit4]   2> 1719633 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
>>  [junit4]   2> 1719634 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
>>  [junit4]   2> 1719650 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
>>  [junit4]   2> 1719651 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
>>  [junit4]   2> 1719683 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
>>  [junit4]   2> 1719684 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
>>  [junit4]   2> 1719694 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
>>  [junit4]   2> 1719695 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
>>  [junit4]   2> 1719706 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
>>  [junit4]   2> 1719707 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
>>  [junit4]   2> 1720156 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1720161 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53694
>>  [junit4]   2> 1720162 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1720163 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1720164 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458
>>  [junit4]   2> 1720164 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458/'
>>  [junit4]   2> 1720208 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458/solr.xml
>>  [junit4]   2> 1720327 T5915 oasc.CoreContainer.<init> New CoreContainer 370136241
>>  [junit4]   2> 1720328 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458/]
>>  [junit4]   2> 1720330 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1720330 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1720331 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1720332 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1720333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1720333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1720334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1720334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1720335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1720350 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1720351 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1720352 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1720352 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>>  [junit4]   2> 1720353 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1720355 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1720360 T5939 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6954bbc7 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1720361 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1720370 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1720374 T5941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f18b9ff name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1720375 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1720381 T5915 oascc.SolrZkClient.makePath makePath: /overseer/queue
>>  [junit4]   2> 1720394 T5915 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>>  [junit4]   2> 1720406 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes
>>  [junit4]   2> 1720412 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53694_od_jjq%2Fcq
>>  [junit4]   2> 1720416 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53694_od_jjq%2Fcq
>>  [junit4]   2> 1720428 T5915 oascc.SolrZkClient.makePath makePath: /overseer_elect
>>  [junit4]   2> 1720436 T5915 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>>  [junit4]   2> 1720451 T5915 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>>  [junit4]   2> 1720460 T5915 oasc.Overseer.start Overseer (id=91100191848005637-127.0.0.1:53694_od_jjq%2Fcq-n_0000000000) starting
>>  [junit4]   2> 1720471 T5915 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>>  [junit4]   2> 1720486 T5943 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>>  [junit4]   2> 1720488 T5915 oascc.SolrZkClient.makePath makePath: /clusterstate.json
>>  [junit4]   2> 1720490 T5943 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
>>  [junit4]   2> 1720497 T5915 oascc.SolrZkClient.makePath makePath: /aliases.json
>>  [junit4]   2> 1720504 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1720515 T5942 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>>  [junit4]   2> 1720523 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1720524 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1720527 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1720532 T5946 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fdc9238 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1720532 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1720536 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1720546 T5915 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
>>  [junit4]   2> 1721004 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1721012 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53698
>>  [junit4]   2> 1721013 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1721013 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1721014 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275
>>  [junit4]   2> 1721014 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/'
>>  [junit4]   2> 1721059 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solr.xml
>>  [junit4]   2> 1721165 T5915 oasc.CoreContainer.<init> New CoreContainer 2108103389
>>  [junit4]   2> 1721165 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/]
>>  [junit4]   2> 1721167 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1721167 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1721167 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1721168 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1721168 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1721168 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1721169 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1721169 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1721169 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1721186 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1721187 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1721187 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1721188 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>>  [junit4]   2> 1721189 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1721192 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1721207 T5957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65d4e39d name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1721207 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1721220 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1721228 T5959 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4db4e5c6 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1721229 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1721243 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1722255 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53698_od_jjq%2Fcq
>>  [junit4]   2> 1722258 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53698_od_jjq%2Fcq
>>  [junit4]   2> 1722266 T5959 oascc.ZkStateReader$3.process Updating live nodes... (2)
>>  [junit4]   2> 1722266 T5941 oascc.ZkStateReader$3.process Updating live nodes... (2)
>>  [junit4]   2> 1722267 T5946 oascc.ZkStateReader$3.process Updating live nodes... (2)
>>  [junit4]   2> 1722281 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1722281 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1722625 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1722630 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53701
>>  [junit4]   2> 1722631 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1722632 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1722632 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011
>>  [junit4]   2> 1722632 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/'
>>  [junit4]   2> 1722668 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solr.xml
>>  [junit4]   2> 1722738 T5915 oasc.CoreContainer.<init> New CoreContainer 1501927636
>>  [junit4]   2> 1722739 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/]
>>  [junit4]   2> 1722741 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1722741 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1722741 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1722743 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1722743 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1722753 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1722753 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1722753 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1722754 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>>  [junit4]   2> 1722754 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1722755 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1722760 T5971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51f4b436 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1722760 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1722767 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1722771 T5973 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b11c2d8 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1722771 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1722782 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1723792 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53701_od_jjq%2Fcq
>>  [junit4]   2> 1723795 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53701_od_jjq%2Fcq
>>  [junit4]   2> 1723805 T5941 oascc.ZkStateReader$3.process Updating live nodes... (3)
>>  [junit4]   2> 1723806 T5946 oascc.ZkStateReader$3.process Updating live nodes... (3)
>>  [junit4]   2> 1723807 T5959 oascc.ZkStateReader$3.process Updating live nodes... (3)
>>  [junit4]   2> 1723807 T5973 oascc.ZkStateReader$3.process Updating live nodes... (3)
>>  [junit4]   2> 1723827 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1723828 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1724210 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1724215 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53704
>>  [junit4]   2> 1724216 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1724216 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1724217 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557
>>  [junit4]   2> 1724217 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/'
>>  [junit4]   2> 1724246 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solr.xml
>>  [junit4]   2> 1724331 T5915 oasc.CoreContainer.<init> New CoreContainer 1311775510
>>  [junit4]   2> 1724332 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/]
>>  [junit4]   2> 1724333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1724333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1724334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1724334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1724334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1724346 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1724347 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1724347 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1724347 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>>  [junit4]   2> 1724348 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1724349 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1724354 T5985 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38426974 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1724354 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1724361 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1724364 T5987 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22116710 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1724365 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1724377 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1725388 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53704_od_jjq%2Fcq
>>  [junit4]   2> 1725391 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53704_od_jjq%2Fcq
>>  [junit4]   2> 1725402 T5946 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1725402 T5941 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1725403 T5959 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1725403 T5987 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1725403 T5973 oascc.ZkStateReader$3.process Updating live nodes... (4)
>>  [junit4]   2> 1725420 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1725421 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1725880 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>>  [junit4]   2> 1725885 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53707
>>  [junit4]   2> 1725886 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>>  [junit4]   2> 1725886 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>>  [junit4]   2> 1725887 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150
>>  [junit4]   2> 1725888 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/'
>>  [junit4]   2> 1725922 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solr.xml
>>  [junit4]   2> 1725993 T5915 oasc.CoreContainer.<init> New CoreContainer 1201695941
>>  [junit4]   2> 1725994 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/]
>>  [junit4]   2> 1725995 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>>  [junit4]   2> 1725995 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>>  [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>>  [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>>  [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>>  [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>>  [junit4]   2> 1725997 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>>  [junit4]   2> 1725997 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>>  [junit4]   2> 1725998 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>>  [junit4]   2> 1726008 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>  [junit4]   2> 1726008 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>>  [junit4]   2> 1726008 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>>  [junit4]   2> 1726009 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>>  [junit4]   2> 1726009 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>>  [junit4]   2> 1726010 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1726014 T5999 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75451e3f name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1726015 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1726022 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>>  [junit4]   2> 1726028 T6001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@509b80a4 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>>  [junit4]   2> 1726028 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>>  [junit4]   2> 1726040 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>>  [junit4]   2> 1727054 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53707_od_jjq%2Fcq
>>  [junit4]   2> 1727056 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53707_od_jjq%2Fcq
>>  [junit4]   2> 1727068 T5959 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1727068 T5946 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1727068 T5987 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1727068 T6001 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1727068 T5941 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1727068 T5973 oascc.ZkStateReader$3.process Updating live nodes... (5)
>>  [junit4]   2> 1727088 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>>  [junit4]   2> 1727088 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>>  [junit4]   2> 1727135 T5953 oasha.CollectionsHandler.handleCreateAction Creating Collection : version=2&numShards=2&action=CREATE&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&replicationFactor=2&wt=javabin&router.name=compositeId
>>  [junit4]   2> 1727138 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1727142 T5943 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
>>  [junit4]   2> 	  "operation":"createcollection",
>>  [junit4]   2> 	  "fromApi":"true",
>>  [junit4]   2> 	  "name":"solrj_collection",
>>  [junit4]   2> 	  "replicationFactor":"2",
>>  [junit4]   2> 	  "collection.configName":"conf1",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "router.field":"myOwnField",
>>  [junit4]   2> 	  "router.name":"compositeId"}
>>  [junit4]   2> 1727142 T5943 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
>>  [junit4]   2> 	  "operation":"createcollection",
>>  [junit4]   2> 	  "fromApi":"true",
>>  [junit4]   2> 	  "name":"solrj_collection",
>>  [junit4]   2> 	  "replicationFactor":"2",
>>  [junit4]   2> 	  "collection.configName":"conf1",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "router.field":"myOwnField",
>>  [junit4]   2> 	  "router.name":"compositeId"}
>>  [junit4]   2> 1727143 T5943 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
>>  [junit4]   2> 1727143 T5943 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
>>  [junit4]   2> 1727156 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1727160 T5942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>>  [junit4]   2> 1727161 T5942 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
>>  [junit4]   2> 1727162 T5942 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
>>  [junit4]   2> 1727169 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1727172 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727172 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727172 T5946 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727172 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727172 T5973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727172 T6001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727257 T5943 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
>>  [junit4]   2> 1727258 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:53698_od_jjq%2Fcq
>>  [junit4]   2> 1727259 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:53707_od_jjq%2Fcq
>>  [junit4]   2> 1727260 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:53704_od_jjq%2Fcq
>>  [junit4]   2> 1727262 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:53701_od_jjq%2Fcq
>>  [junit4]   2> 1727267 T5954 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
>>  [junit4]   2> 1727271 T5979 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
>>  [junit4]   2> 1727273 T5995 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
>>  [junit4]   2> 1727273 T5969 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
>>  [junit4]   2> 1727276 T5954 oasc.ZkController.waitForCoreNodeName look for our core node name
>>  [junit4]   2> 1727277 T5979 oasc.ZkController.waitForCoreNodeName look for our core node name
>>  [junit4]   2> 1727277 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1727279 T5969 oasc.ZkController.waitForCoreNodeName look for our core node name
>>  [junit4]   2> 1727281 T5995 oasc.ZkController.waitForCoreNodeName look for our core node name
>>  [junit4]   2> 1727284 T5942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>>  [junit4]   2> 1727287 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>>  [junit4]   2> 	  "operation":"state",
>>  [junit4]   2> 	  "state":"down",
>>  [junit4]   2> 	  "base_url":"http://127.0.0.1:53698/od_jjq/cq",
>>  [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
>>  [junit4]   2> 	  "roles":null,
>>  [junit4]   2> 	  "node_name":"127.0.0.1:53698_od_jjq%2Fcq",
>>  [junit4]   2> 	  "shard":"shard1",
>>  [junit4]   2> 	  "shard_range":null,
>>  [junit4]   2> 	  "shard_state":"active",
>>  [junit4]   2> 	  "shard_parent":null,
>>  [junit4]   2> 	  "collection":"solrj_collection",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "core_node_name":null}
>>  [junit4]   2> 1727295 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1727300 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>>  [junit4]   2> 	  "operation":"state",
>>  [junit4]   2> 	  "state":"down",
>>  [junit4]   2> 	  "base_url":"http://127.0.0.1:53704/od_jjq/cq",
>>  [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
>>  [junit4]   2> 	  "roles":null,
>>  [junit4]   2> 	  "node_name":"127.0.0.1:53704_od_jjq%2Fcq",
>>  [junit4]   2> 	  "shard":"shard2",
>>  [junit4]   2> 	  "shard_range":null,
>>  [junit4]   2> 	  "shard_state":"active",
>>  [junit4]   2> 	  "shard_parent":null,
>>  [junit4]   2> 	  "collection":"solrj_collection",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "core_node_name":null}
>>  [junit4]   2> 1727310 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1727314 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>>  [junit4]   2> 	  "operation":"state",
>>  [junit4]   2> 	  "state":"down",
>>  [junit4]   2> 	  "base_url":"http://127.0.0.1:53701/od_jjq/cq",
>>  [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
>>  [junit4]   2> 	  "roles":null,
>>  [junit4]   2> 	  "node_name":"127.0.0.1:53701_od_jjq%2Fcq",
>>  [junit4]   2> 	  "shard":"shard2",
>>  [junit4]   2> 	  "shard_range":null,
>>  [junit4]   2> 	  "shard_state":"active",
>>  [junit4]   2> 	  "shard_parent":null,
>>  [junit4]   2> 	  "collection":"solrj_collection",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "core_node_name":null}
>>  [junit4]   2> 1727321 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1727324 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>>  [junit4]   2> 	  "operation":"state",
>>  [junit4]   2> 	  "state":"down",
>>  [junit4]   2> 	  "base_url":"http://127.0.0.1:53707/od_jjq/cq",
>>  [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
>>  [junit4]   2> 	  "roles":null,
>>  [junit4]   2> 	  "node_name":"127.0.0.1:53707_od_jjq%2Fcq",
>>  [junit4]   2> 	  "shard":"shard1",
>>  [junit4]   2> 	  "shard_range":null,
>>  [junit4]   2> 	  "shard_state":"active",
>>  [junit4]   2> 	  "shard_parent":null,
>>  [junit4]   2> 	  "collection":"solrj_collection",
>>  [junit4]   2> 	  "numShards":"2",
>>  [junit4]   2> 	  "core_node_name":null}
>>  [junit4]   2> 1727332 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1727445 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727445 T5973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727446 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727445 T6001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727446 T5946 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1727446 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1728278 T5954 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
>>  [junit4]   2> 1728278 T5979 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
>>  [junit4]   2> 1728279 T5954 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solr.xml
>>  [junit4]   2> 1728279 T5979 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solr.xml
>>  [junit4]   2> 1728280 T5979 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solrj_collection_shard2_replica1
>>  [junit4]   2> 1728280 T5979 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>>  [junit4]   2> 1728281 T5969 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
>>  [junit4]   2> 1728280 T5954 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1
>>  [junit4]   2> 1728281 T5954 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>>  [junit4]   2> 1728282 T5969 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solr.xml
>>  [junit4]   2> 1728282 T5969 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2
>>  [junit4]   2> 1728282 T5969 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>>  [junit4]   2> 1728282 T5995 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
>>  [junit4]   2> 1728284 T5995 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solr.xml
>>  [junit4]   2> 1728284 T5979 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>>  [junit4]   2> 1728284 T5995 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2
>>  [junit4]   2> 1728284 T5954 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>>  [junit4]   2> 1728285 T5954 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>>  [junit4]   2> 1728285 T5995 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>>  [junit4]   2> 1728285 T5969 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>>  [junit4]   2> 1728286 T5969 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>>  [junit4]   2> 1728284 T5979 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>>  [junit4]   2> 1728287 T5995 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>>  [junit4]   2> 1728287 T5995 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>>  [junit4]   2> 1728289 T5954 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/'
>>  [junit4]   2> 1728291 T5995 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/'
>>  [junit4]   2> 1728292 T5969 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/'
>>  [junit4]   2> 1728295 T5979 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solrj_collection_shard2_replica1/'
>>  [junit4]   2> 1728399 T5954 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>>  [junit4]   2> 1728413 T5995 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>>  [junit4]   2> 1728426 T5969 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>>  [junit4]   2> 1728492 T5979 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>>  [junit4]   2> 1728547 T5969 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>>  [junit4]   2> 1728550 T5995 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>>  [junit4]   2> 1728554 T5969 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>>  [junit4]   2> 1728574 T5995 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>>  [junit4]   2> 1728577 T5969 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
>>  [junit4]   2> 1728610 T5995 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
>>  [junit4]   2> 1728640 T5954 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>>  [junit4]   2> 1728652 T5979 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>>  [junit4]   2> 1728653 T5954 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>>  [junit4]   2> 1728663 T5954 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
>>  [junit4]   2> 1728663 T5979 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>>  [junit4]   2> 1728715 T5979 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
>>  [junit4]   2> 1730003 T5954 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1730003 T5954 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1730004 T5954 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>>  [junit4]   2> 1730012 T5954 oass.IndexSchema.readSchema default search field in schema is text
>>  [junit4]   2> 1730018 T5954 oass.IndexSchema.readSchema unique key field: id
>>  [junit4]   2> 1730071 T5969 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1730071 T5969 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1730073 T5969 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>>  [junit4]   2> 1730090 T5969 oass.IndexSchema.readSchema default search field in schema is text
>>  [junit4]   2> 1730097 T5969 oass.IndexSchema.readSchema unique key field: id
>>  [junit4]   2> 1730128 T5995 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1730128 T5995 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1730129 T5995 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>>  [junit4]   2> 1730129 T5954 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1730137 T5969 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1730141 T5954 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1730153 T5995 oass.IndexSchema.readSchema default search field in schema is text
>>  [junit4]   2> 1730158 T5995 oass.IndexSchema.readSchema unique key field: id
>>  [junit4]   2> 1730166 T5969 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1730176 T5954 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1730180 T5969 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1730181 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1730182 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1730182 T5954 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1730182 T5979 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1730184 T5979 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>>  [junit4]   2> 1730184 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1730184 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1730185 T5969 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1730186 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1730186 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1730187 T5954 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>>  [junit4]   2> 1730187 T5954 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/
>>  [junit4]   2> 1730188 T5954 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@589e0cdc
>>  [junit4]   2> 1730185 T5979 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>>  [junit4]   2> 1730189 T5954 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data
>>  [junit4]   2> 1730189 T5954 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/index/
>>  [junit4]   2> 1730190 T5954 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
>>  [junit4]   2> 1730190 T5954 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/index
>>  [junit4]   2> 1730191 T5954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
>>  [junit4]   2> 1730192 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1730192 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1730192 T5969 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>>  [junit4]   2> 1730193 T5969 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/
>>  [junit4]   2> 1730193 T5969 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@589e0cdc
>>  [junit4]   2> 1730194 T5969 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data
>>  [junit4]   2> 1730195 T5969 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/index/
>>  [junit4]   2> 1730195 T5969 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
>>  [junit4]   2> 1730197 T5969 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/index
>>  [junit4]   2> 1730197 T5969 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
>>  [junit4]   2> 1730199 T5954 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@247327d5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@392d8b48),segFN=segments_1,generation=1}
>>  [junit4]   2> 1730199 T5954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1730199 T5969 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@434ea2f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4d5719cb),segFN=segments_1,generation=1}
>>  [junit4]   2> 1730200 T5969 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1730200 T5979 oass.IndexSchema.readSchema default search field in schema is text
>>  [junit4]   2> 1730204 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>>  [junit4]   2> 1730204 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1730205 T5969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1730206 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1730207 T5969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1730207 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>>  [junit4]   2> 1730208 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1730209 T5969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1730209 T5969 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>>  [junit4]   2> 1730204 T5979 oass.IndexSchema.readSchema unique key field: id
>>  [junit4]   2> 1730210 T5969 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>>  [junit4]   2> 1730212 T5969 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>>  [junit4]   2> 1730213 T5969 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>>  [junit4]   2> 1730213 T5969 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>>  [junit4]   2> 1730214 T5969 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>>  [junit4]   2> 1730215 T5969 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>>  [junit4]   2> 1730209 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>>  [junit4]   2> 1730216 T5969 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>>  [junit4]   2> 1730216 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1730217 T5954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1730217 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1730218 T5954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1730219 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>>  [junit4]   2> 1730220 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1730220 T5954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1730221 T5954 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>>  [junit4]   2> 1730222 T5954 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>>  [junit4]   2> 1730222 T5954 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>>  [junit4]   2> 1730223 T5954 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>>  [junit4]   2> 1730223 T5954 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>>  [junit4]   2> 1730224 T5954 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>>  [junit4]   2> 1730227 T5954 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>>  [junit4]   2> 1730226 T5969 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>>  [junit4]   2> 1730228 T5954 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>>  [junit4]   2> 1730235 T5954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>>  [junit4]   2> 1730240 T5969 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>>  [junit4]   2> 1730241 T5969 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>>  [junit4]   2> 1730244 T5969 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=43, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6018786168657202]
>>  [junit4]   2> 1730245 T5969 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@434ea2f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4d5719cb),segFN=segments_1,generation=1}
>>  [junit4]   2> 1730245 T5969 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1730245 T5969 oass.SolrIndexSearcher.<init> Opening Searcher@682747bb main
>>  [junit4]   2> 1730246 T5954 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>>  [junit4]   2> 1730249 T5954 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>>  [junit4]   2> 1730249 T5995 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1730251 T5954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=43, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6018786168657202]
>>  [junit4]   2> 1730253 T5954 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@247327d5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@392d8b48),segFN=segments_1,generation=1}
>>  [junit4]   2> 1730253 T5954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1730254 T5954 oass.SolrIndexSearcher.<init> Opening Searcher@7487fd38 main
>>  [junit4]   2> 1730260 T6008 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@7487fd38 main{StandardDirectoryReader(segments_1:1:nrt)}
>>  [junit4]   2> 1730263 T5995 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1730263 T5954 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
>>  [junit4]   2> 1730264 T5954 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:http://127.0.0.1:53698/od_jjq/cq collection:solrj_collection shard:shard1
>>  [junit4]   2> 1730264 T6007 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@682747bb main{StandardDirectoryReader(segments_1:1:nrt)}
>>  [junit4]   2> 1730268 T5954 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
>>  [junit4]   2> 1730288 T5995 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1730288 T5969 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
>>  [junit4]   2> 1730289 T5969 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica2 address:http://127.0.0.1:53701/od_jjq/cq collection:solrj_collection shard:shard2
>>  [junit4]   2> 1730292 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1730293 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1730294 T5995 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1730300 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1730301 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1730302 T5995 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>>  [junit4]   2> 1730302 T5995 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/
>>  [junit4]   2> 1730303 T5995 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@589e0cdc
>>  [junit4]   2> 1730294 T5969 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
>>  [junit4]   2> 1730308 T5995 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data
>>  [junit4]   2> 1730309 T5995 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/index/
>>  [junit4]   2> 1730310 T5995 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
>>  [junit4]   2> 1730311 T5995 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/index
>>  [junit4]   2> 1730312 T5995 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
>>  [junit4]   2> 1730319 T5995 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>>  [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@858afaf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@659dfd2a),segFN=segments_1,generation=1}
>>  [junit4]   2> 1730320 T5995 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>>  [junit4]   2> 1730322 T5979 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1730325 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>>  [junit4]   2> 1730331 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>>  [junit4]   2> 1730332 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1730333 T5995 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>>  [junit4]   2> 1730333 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1730334 T5995 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>>  [junit4]   2> 1730335 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>>  [junit4]   2> 1730336 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1730338 T5995 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>>  [junit4]   2> 1730339 T5995 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>>  [junit4]   2> 1730339 T5979 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>>  [junit4]   2> 1730340 T5995 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>>  [junit4]   2> 1730341 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1730343 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1730343 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1730344 T5995 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>>  [junit4]   2> 1730344 T5995 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>>  [junit4]   2> 1730346 T5995 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>>  [junit4]   2> 1730346 T5954 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=-4
>>  [junit4]   2> 1730349 T5954 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
>>  [junit4]   2> 1730349 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>>  [junit4]   2> ASYNC  NEW_CORE C4181 name=solrj_collection_shard1_replica1 org.apache.solr.core.SolrCore@46d7249d url=http://127.0.0.1:53698/od_jjq/cq/solrj_collection_shard1_replica1 node=127.0.0.1:53698_od_jjq%2Fcq C4181_STATE=coll:solrj_collection core:solrj_collection_shard1_replica1 props:{state=down, base_url=http://127.0.0.1:53698/od_jjq/cq, core=solrj_collection_shard1_replica1, node_name=127.0.0.1:53698_od_jjq%2Fcq}
>>  [junit4]   2> 1730350 T5954 C4181 P53698 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53698/od_jjq/cq/solrj_collection_shard1_replica1/
>>  [junit4]   2> 1730350 T5954 C4181 P53698 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=http://127.0.0.1:53698/od_jjq/cq START replicas=[http://127.0.0.1:53707/od_jjq/cq/solrj_collection_shard1_replica2/] nUpdates=100
>>  [junit4]   2> 1730353 T5995 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>>  [junit4]   2> 1730354 T5995 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>>  [junit4]   2> 1730356 T5942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>>  [junit4]   2> 1730354 T5979 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1730358 T5995 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>>  [junit4]   2> 1730361 T5969 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
>>  [junit4]   2> 1730367 T5995 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>>  [junit4]   2> 1730369 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1730372 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1730373 T5979 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>>  [junit4]   2> 1730367 T5954 C4181 P53698 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=http://127.0.0.1:53698/od_jjq/cq DONE.  We have no versions.  sync failed.
>>  [junit4]   2> 1730374 T5954 C4181 P53698 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
>>  [junit4]   2> 1730374 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
>>  [junit4]   2> 1730379 T5946 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1730379 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53698/od_jjq/cq/solrj_collection_shard1_replica1/ shard1
>>  [junit4]   2> 1730375 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>>  [junit4]   2> 1730384 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1730382 T6001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1730381 T5973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1730380 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1730385 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>>  [junit4]   2> 1730384 T5954 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard1
>>  [junit4]   2> 1730391 T5969 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=-9
>>  [junit4]   2> 1730395 T5969 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
>>  [junit4]   2> 1730396 T5969 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>>  [junit4]   2> 1730396 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>>  [junit4]   2> 1730397 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>>  [junit4]   2> 1730397 T5979 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>>  [junit4]   2> 1730395 T5995 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>>  [junit4]   2> 1730399 T5995 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>>  [junit4]   2> 1730398 T5979 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/wo
>> 
>> [...truncated too long message...]
>> 
>> )
>>  [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
>>  [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>>  [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>>  [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>>  [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>>  [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>>  [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>>  [junit4]   2> 	
>>  [junit4]   2> 2162194 T5987 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/nodes_used_collection/leader_elect/shard1/election/91100191848005644-core_node1-n_0000000001
>>  [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>>  [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>  [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
>>  [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
>>  [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>>  [junit4]   2> 
>>  [junit4]   2> 2162195 T5987 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>>  [junit4]   2> 2162195 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
>>  [junit4]   2> 2162195 T5987 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>>  [junit4]   2> 2162196 T5987 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
>>  [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>>  [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>  [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>>  [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>>  [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>>  [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>>  [junit4]   2> 
>>  [junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_IQ, timezone=Etc/UTC
>>  [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=1,free=107685104,total=385253376
>>  [junit4]   2> NOTE: All tests run in this JVM: [PluginInfoTest, QueryElevationComponentTest, ZkCLITest, TestCollationField, CurrencyFieldXmlFileTest, BadCopyFieldTest, TestSolrXmlPersistence, TestPartialUpdateDeduplication, TestManagedSchema, WordBreakSolrSpellCheckerTest, RAMDirectoryFactoryTest, TestDistributedMissingSort, TestJmxIntegration, TestElisionMultitermQuery, SpellCheckComponentTest, ZkSolrClientTest, HdfsBasicDistributedZk2Test, TestArbitraryIndexDir, TestSimpleQParserPlugin, TestRandomDVFaceting, TestSchemaSimilarityResource, OutputWriterTest, TestQueryTypes, CopyFieldTest, TestStressReorder, DistanceFunctionTest, SuggestComponentTest, MigrateRouteKeyTest, DateFieldTest, DistributedDebugComponentTest, TestCloudManagedSchemaAddField, FieldAnalysisRequestHandlerTest, TestFileDictionaryLookup, TestDocBasedVersionConstraints, LeaderElectionIntegrationTest, TestCollapseQParserPlugin, TestMultiCoreConfBootstrap, QueryResultKeyTest, TestDocSet, TestLMDirichletSimilarityFactory, TestCSVLoader, TestRTGBase, TestSchemaVersionResource, FileUtilsTest, TriLevelCompositeIdRoutingTest, ParsingFieldUpdateProcessorsTest, TestCodecSupport, DistributedTermsComponentTest, TestCustomSort, ConnectionManagerTest, RequiredFieldsTest, TestQueryUtils, TestReplicationHandler, TestSolrXMLSerializer, TestUtils, UpdateRequestProcessorFactoryTest, TestFastWriter, TestPHPSerializedResponseWriter, OpenExchangeRatesOrgProviderTest, TestFoldingMultitermQuery, TestNonNRTOpen, DOMUtilTest, CSVRequestHandlerTest, TestIndexingPerformance, DocumentAnalysisRequestHandlerTest, TestLuceneMatchVersion, DocValuesTest, TestJoin, TestSchemaNameResource, TestBadConfig, CoreAdminCreateDiscoverTest, HighlighterConfigTest, TestPerFieldSimilarity, BlockCacheTest, SolrIndexSplitterTest, URLClassifyProcessorTest, SchemaVersionSpecificBehaviorTest, QueryParsingTest, CursorMarkTest, SimplePostToolTest, TestNumberUtils, EnumFieldTest, ChaosMonkeySafeLeaderTest, DirectUpdateHandlerOptimizeTest, TestCollationFieldDocValues, TestFuzzyAnalyzedSuggestions, TestQuerySenderListener, HdfsRecoveryZkTest, CoreMergeIndexesAdminHandlerTest, ClusterStateTest, TestSolr4Spatial, TestSystemIdResolver, TestStressLucene, SolrCoreCheckLockOnStartupTest, TestManagedSchemaFieldResource, IndexSchemaTest, TestWordDelimiterFilterFactory, TestOmitPositions, TermVectorComponentTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestIndexSearcher, TestPseudoReturnFields, HdfsChaosMonkeySafeLeaderTest, TestLazyCores, DateMathParserTest, TestAtomicUpdateErrorCases, TestLFUCache, TestDefaultSimilarityFactory, TestPostingsSolrHighlighter, SimpleFacetsTest, TestImplicitCoreProperties, DeleteReplicaTest, TestSerializedLuceneMatchVersion, TestHighFrequencyDictionaryFactory, ChangedSchemaMergeTest, TestAnalyzedSuggestions, CursorPagingTest, PingRequestHandlerTest, TestSuggestSpellingConverter, TestDFRSimilarityFactory, TestSolrDeletionPolicy2, PrimUtilsTest, SignatureUpdateProcessorFactoryTest, TestStressUserVersions, TestDistribDocBasedVersion, PrimitiveFieldTypeTest, ShardRoutingCustomTest, TestCloudManagedSchema, TestDynamicFieldCollectionResource, AutoCommitTest, SolrPluginUtilsTest, BlockDirectoryTest, SolrIndexConfigTest, ShowFileRequestHandlerTest, JSONWriterTest, TestStressVersions, SpatialFilterTest, BadIndexSchemaTest, ShardRoutingTest, TestShardHandlerFactory, SuggesterTest, TestBM25SimilarityFactory, DirectUpdateHandlerTest, DistribCursorPagingTest, FieldMutatingUpdateProcessorTest, RecoveryZkTest, TestNoOpRegenerator, TestSolrDeletionPolicy1, BasicDistributedZkTest, TestBinaryResponseWriter, TestRandomMergePolicy, ExternalFileFieldSortTest, OpenCloseCoreStressTest, PolyFieldTest, BasicDistributedZk2Test, TestFreeTextSuggestions, JsonLoaderTest, TestAddFieldRealTimeGet, UniqFieldsUpdateProcessorFactoryTest, TestAnalyzeInfixSuggestions, TestStressRecovery, TestBinaryField, IndexSchemaRuntimeFieldTest, TestSolrQueryParser, HdfsBasicDistributedZkTest, ZkNodePropsTest, BadComponentTest, UUIDFieldTest, AssignTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
>>  [junit4] Completed in 443.02s, 1 test, 1 failure <<< FAILURES!
>> 
>> [...truncated 539 lines...]
>> BUILD FAILED
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1540 tests, 1 failure, 40 ignored (6 assumptions)
>> 
>> Total time: 85 minutes 49 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1210 - Still Failing!

Posted by Mark Miller <ma...@gmail.com>.
Looks like it’s stuck in a socket read. I’ll turn the tests distributed so timeout back on.

- Mark

On Jan 18, 2014, at 4:22 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1210/
> Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
> 
> 1 tests failed.
> FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
> 
> Error Message:
> There are still nodes recoverying - waited for 330 seconds
> 
> Stack Trace:
> java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
> 	at __randomizedtesting.SeedInfo.seed([E01973DE033FE27B:61FFFDC674608247]:0)
> 	at org.junit.Assert.fail(Assert.java:93)
> 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
> 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
> 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
> 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:772)
> 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:196)
> 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
> 	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> 	at java.lang.Thread.run(Thread.java:744)
> 
> 
> 
> 
> Build Log:
> [...truncated 10429 lines...]
>   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
>   [junit4]   2> 1719213 T5915 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /od_jjq/cq
>   [junit4]   2> 1719222 T5915 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
>   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1390078610952
>   [junit4]   2> 1719224 T5915 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>   [junit4]   2> 1719225 T5916 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>   [junit4]   2> 1719327 T5915 oasc.ZkTestServer.run start zk server on port:53689
>   [junit4]   2> 1719329 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1719338 T5922 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b2bf8c1 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1719338 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1719339 T5915 oascc.SolrZkClient.makePath makePath: /solr
>   [junit4]   2> 1719353 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1719358 T5924 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31e8fc2a name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1719359 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1719359 T5915 oascc.SolrZkClient.makePath makePath: /collections/collection1
>   [junit4]   2> 1719375 T5915 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>   [junit4]   2> 1719384 T5915 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>   [junit4]   2> 1719393 T5915 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>   [junit4]   2> 1719403 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
>   [junit4]   2> 1719404 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>   [junit4]   2> 1719421 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
>   [junit4]   2> 1719422 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>   [junit4]   2> 1719432 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 1719433 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 1719444 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>   [junit4]   2> 1719445 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>   [junit4]   2> 1719456 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>   [junit4]   2> 1719457 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>   [junit4]   2> 1719469 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>   [junit4]   2> 1719470 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>   [junit4]   2> 1719480 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>   [junit4]   2> 1719481 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>   [junit4]   2> 1719492 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 1719493 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 1719503 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 1719504 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 1719513 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>   [junit4]   2> 1719514 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>   [junit4]   2> 1719524 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>   [junit4]   2> 1719526 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>   [junit4]   2> Using old style solr.xml
>   [junit4]   2> 1719542 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1719547 T5926 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66930f41 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1719547 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1719548 T5915 oascc.SolrZkClient.makePath makePath: /solr
>   [junit4]   2> 1719558 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1719562 T5928 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3239c4d9 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1719562 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1719564 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
>   [junit4]   2> 1719565 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
>   [junit4]   2> 1719576 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
>   [junit4]   2> 1719578 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
>   [junit4]   2> 1719588 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 1719589 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 1719600 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
>   [junit4]   2> 1719600 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
>   [junit4]   2> 1719611 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
>   [junit4]   2> 1719612 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
>   [junit4]   2> 1719622 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
>   [junit4]   2> 1719624 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
>   [junit4]   2> 1719633 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
>   [junit4]   2> 1719634 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
>   [junit4]   2> 1719650 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 1719651 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 1719683 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
>   [junit4]   2> 1719684 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
>   [junit4]   2> 1719694 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
>   [junit4]   2> 1719695 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
>   [junit4]   2> 1719706 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
>   [junit4]   2> 1719707 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
>   [junit4]   2> 1720156 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1720161 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53694
>   [junit4]   2> 1720162 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1720163 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1720164 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458
>   [junit4]   2> 1720164 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458/'
>   [junit4]   2> 1720208 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458/solr.xml
>   [junit4]   2> 1720327 T5915 oasc.CoreContainer.<init> New CoreContainer 370136241
>   [junit4]   2> 1720328 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458/]
>   [junit4]   2> 1720330 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1720330 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1720331 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1720332 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1720333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1720333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1720334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1720334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1720335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1720350 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1720351 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1720352 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1720352 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>   [junit4]   2> 1720353 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1720355 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1720360 T5939 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6954bbc7 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1720361 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1720370 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1720374 T5941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f18b9ff name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1720375 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1720381 T5915 oascc.SolrZkClient.makePath makePath: /overseer/queue
>   [junit4]   2> 1720394 T5915 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>   [junit4]   2> 1720406 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes
>   [junit4]   2> 1720412 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53694_od_jjq%2Fcq
>   [junit4]   2> 1720416 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53694_od_jjq%2Fcq
>   [junit4]   2> 1720428 T5915 oascc.SolrZkClient.makePath makePath: /overseer_elect
>   [junit4]   2> 1720436 T5915 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>   [junit4]   2> 1720451 T5915 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>   [junit4]   2> 1720460 T5915 oasc.Overseer.start Overseer (id=91100191848005637-127.0.0.1:53694_od_jjq%2Fcq-n_0000000000) starting
>   [junit4]   2> 1720471 T5915 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>   [junit4]   2> 1720486 T5943 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>   [junit4]   2> 1720488 T5915 oascc.SolrZkClient.makePath makePath: /clusterstate.json
>   [junit4]   2> 1720490 T5943 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
>   [junit4]   2> 1720497 T5915 oascc.SolrZkClient.makePath makePath: /aliases.json
>   [junit4]   2> 1720504 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1720515 T5942 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>   [junit4]   2> 1720523 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1720524 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1720527 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1720532 T5946 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fdc9238 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1720532 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1720536 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1720546 T5915 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
>   [junit4]   2> 1721004 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1721012 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53698
>   [junit4]   2> 1721013 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1721013 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1721014 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275
>   [junit4]   2> 1721014 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/'
>   [junit4]   2> 1721059 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solr.xml
>   [junit4]   2> 1721165 T5915 oasc.CoreContainer.<init> New CoreContainer 2108103389
>   [junit4]   2> 1721165 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/]
>   [junit4]   2> 1721167 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1721167 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1721167 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1721168 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1721168 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1721168 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1721169 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1721169 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1721169 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1721186 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1721187 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1721187 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1721188 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>   [junit4]   2> 1721189 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1721192 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1721207 T5957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65d4e39d name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1721207 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1721220 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1721228 T5959 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4db4e5c6 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1721229 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1721243 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1722255 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53698_od_jjq%2Fcq
>   [junit4]   2> 1722258 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53698_od_jjq%2Fcq
>   [junit4]   2> 1722266 T5959 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 1722266 T5941 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 1722267 T5946 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 1722281 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1722281 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1722625 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1722630 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53701
>   [junit4]   2> 1722631 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1722632 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1722632 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011
>   [junit4]   2> 1722632 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/'
>   [junit4]   2> 1722668 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solr.xml
>   [junit4]   2> 1722738 T5915 oasc.CoreContainer.<init> New CoreContainer 1501927636
>   [junit4]   2> 1722739 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/]
>   [junit4]   2> 1722741 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1722741 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1722741 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1722743 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1722743 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1722753 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1722753 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1722753 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1722754 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>   [junit4]   2> 1722754 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1722755 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1722760 T5971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51f4b436 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1722760 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1722767 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1722771 T5973 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b11c2d8 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1722771 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1722782 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1723792 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53701_od_jjq%2Fcq
>   [junit4]   2> 1723795 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53701_od_jjq%2Fcq
>   [junit4]   2> 1723805 T5941 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 1723806 T5946 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 1723807 T5959 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 1723807 T5973 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 1723827 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1723828 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1724210 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1724215 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53704
>   [junit4]   2> 1724216 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1724216 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1724217 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557
>   [junit4]   2> 1724217 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/'
>   [junit4]   2> 1724246 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solr.xml
>   [junit4]   2> 1724331 T5915 oasc.CoreContainer.<init> New CoreContainer 1311775510
>   [junit4]   2> 1724332 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/]
>   [junit4]   2> 1724333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1724333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1724334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1724334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1724334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1724346 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1724347 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1724347 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1724347 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>   [junit4]   2> 1724348 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1724349 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1724354 T5985 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38426974 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1724354 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1724361 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1724364 T5987 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22116710 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1724365 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1724377 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1725388 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53704_od_jjq%2Fcq
>   [junit4]   2> 1725391 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53704_od_jjq%2Fcq
>   [junit4]   2> 1725402 T5946 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1725402 T5941 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1725403 T5959 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1725403 T5987 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1725403 T5973 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 1725420 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1725421 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1725880 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 1725885 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53707
>   [junit4]   2> 1725886 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 1725886 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1725887 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150
>   [junit4]   2> 1725888 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/'
>   [junit4]   2> 1725922 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solr.xml
>   [junit4]   2> 1725993 T5915 oasc.CoreContainer.<init> New CoreContainer 1201695941
>   [junit4]   2> 1725994 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/]
>   [junit4]   2> 1725995 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
>   [junit4]   2> 1725995 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 1725997 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 1725997 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 1725998 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 1726008 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 1726008 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 1726008 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 1726009 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
>   [junit4]   2> 1726009 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 1726010 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1726014 T5999 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75451e3f name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1726015 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1726022 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1726028 T6001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@509b80a4 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1726028 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 1726040 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 1727054 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53707_od_jjq%2Fcq
>   [junit4]   2> 1727056 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53707_od_jjq%2Fcq
>   [junit4]   2> 1727068 T5959 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1727068 T5946 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1727068 T5987 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1727068 T6001 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1727068 T5941 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1727068 T5973 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 1727088 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 1727088 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 1727135 T5953 oasha.CollectionsHandler.handleCreateAction Creating Collection : version=2&numShards=2&action=CREATE&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&replicationFactor=2&wt=javabin&router.name=compositeId
>   [junit4]   2> 1727138 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1727142 T5943 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
>   [junit4]   2> 	  "operation":"createcollection",
>   [junit4]   2> 	  "fromApi":"true",
>   [junit4]   2> 	  "name":"solrj_collection",
>   [junit4]   2> 	  "replicationFactor":"2",
>   [junit4]   2> 	  "collection.configName":"conf1",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "router.field":"myOwnField",
>   [junit4]   2> 	  "router.name":"compositeId"}
>   [junit4]   2> 1727142 T5943 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
>   [junit4]   2> 	  "operation":"createcollection",
>   [junit4]   2> 	  "fromApi":"true",
>   [junit4]   2> 	  "name":"solrj_collection",
>   [junit4]   2> 	  "replicationFactor":"2",
>   [junit4]   2> 	  "collection.configName":"conf1",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "router.field":"myOwnField",
>   [junit4]   2> 	  "router.name":"compositeId"}
>   [junit4]   2> 1727143 T5943 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
>   [junit4]   2> 1727143 T5943 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
>   [junit4]   2> 1727156 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1727160 T5942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 1727161 T5942 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
>   [junit4]   2> 1727162 T5942 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
>   [junit4]   2> 1727169 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1727172 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727172 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727172 T5946 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727172 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727172 T5973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727172 T6001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727257 T5943 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
>   [junit4]   2> 1727258 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:53698_od_jjq%2Fcq
>   [junit4]   2> 1727259 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:53707_od_jjq%2Fcq
>   [junit4]   2> 1727260 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:53704_od_jjq%2Fcq
>   [junit4]   2> 1727262 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:53701_od_jjq%2Fcq
>   [junit4]   2> 1727267 T5954 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
>   [junit4]   2> 1727271 T5979 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
>   [junit4]   2> 1727273 T5995 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
>   [junit4]   2> 1727273 T5969 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
>   [junit4]   2> 1727276 T5954 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 1727277 T5979 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 1727277 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1727279 T5969 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 1727281 T5995 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 1727284 T5942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 1727287 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:53698/od_jjq/cq",
>   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:53698_od_jjq%2Fcq",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"solrj_collection",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 1727295 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1727300 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:53704/od_jjq/cq",
>   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:53704_od_jjq%2Fcq",
>   [junit4]   2> 	  "shard":"shard2",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"solrj_collection",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 1727310 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1727314 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:53701/od_jjq/cq",
>   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:53701_od_jjq%2Fcq",
>   [junit4]   2> 	  "shard":"shard2",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"solrj_collection",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 1727321 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1727324 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:53707/od_jjq/cq",
>   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:53707_od_jjq%2Fcq",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"solrj_collection",
>   [junit4]   2> 	  "numShards":"2",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 1727332 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1727445 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727445 T5973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727446 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727445 T6001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727446 T5946 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1727446 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1728278 T5954 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
>   [junit4]   2> 1728278 T5979 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
>   [junit4]   2> 1728279 T5954 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solr.xml
>   [junit4]   2> 1728279 T5979 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solr.xml
>   [junit4]   2> 1728280 T5979 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solrj_collection_shard2_replica1
>   [junit4]   2> 1728280 T5979 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>   [junit4]   2> 1728281 T5969 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
>   [junit4]   2> 1728280 T5954 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1
>   [junit4]   2> 1728281 T5954 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>   [junit4]   2> 1728282 T5969 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solr.xml
>   [junit4]   2> 1728282 T5969 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2
>   [junit4]   2> 1728282 T5969 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>   [junit4]   2> 1728282 T5995 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
>   [junit4]   2> 1728284 T5995 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solr.xml
>   [junit4]   2> 1728284 T5979 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 1728284 T5995 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2
>   [junit4]   2> 1728284 T5954 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 1728285 T5954 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>   [junit4]   2> 1728285 T5995 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
>   [junit4]   2> 1728285 T5969 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 1728286 T5969 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>   [junit4]   2> 1728284 T5979 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>   [junit4]   2> 1728287 T5995 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 1728287 T5995 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
>   [junit4]   2> 1728289 T5954 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/'
>   [junit4]   2> 1728291 T5995 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/'
>   [junit4]   2> 1728292 T5969 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/'
>   [junit4]   2> 1728295 T5979 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solrj_collection_shard2_replica1/'
>   [junit4]   2> 1728399 T5954 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 1728413 T5995 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 1728426 T5969 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 1728492 T5979 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 1728547 T5969 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 1728550 T5995 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 1728554 T5969 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 1728574 T5995 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 1728577 T5969 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
>   [junit4]   2> 1728610 T5995 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
>   [junit4]   2> 1728640 T5954 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 1728652 T5979 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 1728653 T5954 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 1728663 T5954 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
>   [junit4]   2> 1728663 T5979 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 1728715 T5979 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
>   [junit4]   2> 1730003 T5954 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1730003 T5954 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1730004 T5954 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 1730012 T5954 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 1730018 T5954 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 1730071 T5969 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1730071 T5969 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1730073 T5969 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 1730090 T5969 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 1730097 T5969 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 1730128 T5995 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1730128 T5995 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1730129 T5995 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 1730129 T5954 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1730137 T5969 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1730141 T5954 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1730153 T5995 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 1730158 T5995 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 1730166 T5969 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1730176 T5954 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1730180 T5969 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1730181 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1730182 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1730182 T5954 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1730182 T5979 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1730184 T5979 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 1730184 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1730184 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1730185 T5969 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1730186 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1730186 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1730187 T5954 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 1730187 T5954 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/
>   [junit4]   2> 1730188 T5954 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@589e0cdc
>   [junit4]   2> 1730185 T5979 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 1730189 T5954 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data
>   [junit4]   2> 1730189 T5954 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/index/
>   [junit4]   2> 1730190 T5954 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 1730190 T5954 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/index
>   [junit4]   2> 1730191 T5954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
>   [junit4]   2> 1730192 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1730192 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1730192 T5969 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 1730193 T5969 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/
>   [junit4]   2> 1730193 T5969 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@589e0cdc
>   [junit4]   2> 1730194 T5969 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data
>   [junit4]   2> 1730195 T5969 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/index/
>   [junit4]   2> 1730195 T5969 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 1730197 T5969 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/index
>   [junit4]   2> 1730197 T5969 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
>   [junit4]   2> 1730199 T5954 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@247327d5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@392d8b48),segFN=segments_1,generation=1}
>   [junit4]   2> 1730199 T5954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1730199 T5969 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@434ea2f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4d5719cb),segFN=segments_1,generation=1}
>   [junit4]   2> 1730200 T5969 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1730200 T5979 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 1730204 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 1730204 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1730205 T5969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1730206 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1730207 T5969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1730207 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 1730208 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1730209 T5969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1730209 T5969 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 1730204 T5979 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 1730210 T5969 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 1730212 T5969 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 1730213 T5969 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 1730213 T5969 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 1730214 T5969 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 1730215 T5969 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 1730209 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 1730216 T5969 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 1730216 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1730217 T5954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1730217 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1730218 T5954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1730219 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 1730220 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1730220 T5954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1730221 T5954 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 1730222 T5954 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 1730222 T5954 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 1730223 T5954 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 1730223 T5954 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 1730224 T5954 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 1730227 T5954 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 1730226 T5969 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 1730228 T5954 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 1730235 T5954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 1730240 T5969 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 1730241 T5969 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 1730244 T5969 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=43, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6018786168657202]
>   [junit4]   2> 1730245 T5969 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@434ea2f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4d5719cb),segFN=segments_1,generation=1}
>   [junit4]   2> 1730245 T5969 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1730245 T5969 oass.SolrIndexSearcher.<init> Opening Searcher@682747bb main
>   [junit4]   2> 1730246 T5954 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 1730249 T5954 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 1730249 T5995 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1730251 T5954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=43, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6018786168657202]
>   [junit4]   2> 1730253 T5954 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@247327d5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@392d8b48),segFN=segments_1,generation=1}
>   [junit4]   2> 1730253 T5954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1730254 T5954 oass.SolrIndexSearcher.<init> Opening Searcher@7487fd38 main
>   [junit4]   2> 1730260 T6008 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@7487fd38 main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 1730263 T5995 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1730263 T5954 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
>   [junit4]   2> 1730264 T5954 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:http://127.0.0.1:53698/od_jjq/cq collection:solrj_collection shard:shard1
>   [junit4]   2> 1730264 T6007 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@682747bb main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 1730268 T5954 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
>   [junit4]   2> 1730288 T5995 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1730288 T5969 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
>   [junit4]   2> 1730289 T5969 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica2 address:http://127.0.0.1:53701/od_jjq/cq collection:solrj_collection shard:shard2
>   [junit4]   2> 1730292 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1730293 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1730294 T5995 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1730300 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1730301 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1730302 T5995 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 1730302 T5995 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/
>   [junit4]   2> 1730303 T5995 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@589e0cdc
>   [junit4]   2> 1730294 T5969 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
>   [junit4]   2> 1730308 T5995 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data
>   [junit4]   2> 1730309 T5995 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/index/
>   [junit4]   2> 1730310 T5995 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 1730311 T5995 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/index
>   [junit4]   2> 1730312 T5995 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
>   [junit4]   2> 1730319 T5995 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@858afaf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@659dfd2a),segFN=segments_1,generation=1}
>   [junit4]   2> 1730320 T5995 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 1730322 T5979 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1730325 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>   [junit4]   2> 1730331 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 1730332 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1730333 T5995 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 1730333 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1730334 T5995 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 1730335 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 1730336 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1730338 T5995 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 1730339 T5995 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 1730339 T5979 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 1730340 T5995 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 1730341 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1730343 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1730343 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1730344 T5995 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 1730344 T5995 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 1730346 T5995 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 1730346 T5954 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=-4
>   [junit4]   2> 1730349 T5954 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
>   [junit4]   2> 1730349 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> ASYNC  NEW_CORE C4181 name=solrj_collection_shard1_replica1 org.apache.solr.core.SolrCore@46d7249d url=http://127.0.0.1:53698/od_jjq/cq/solrj_collection_shard1_replica1 node=127.0.0.1:53698_od_jjq%2Fcq C4181_STATE=coll:solrj_collection core:solrj_collection_shard1_replica1 props:{state=down, base_url=http://127.0.0.1:53698/od_jjq/cq, core=solrj_collection_shard1_replica1, node_name=127.0.0.1:53698_od_jjq%2Fcq}
>   [junit4]   2> 1730350 T5954 C4181 P53698 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53698/od_jjq/cq/solrj_collection_shard1_replica1/
>   [junit4]   2> 1730350 T5954 C4181 P53698 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=http://127.0.0.1:53698/od_jjq/cq START replicas=[http://127.0.0.1:53707/od_jjq/cq/solrj_collection_shard1_replica2/] nUpdates=100
>   [junit4]   2> 1730353 T5995 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 1730354 T5995 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 1730356 T5942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 1730354 T5979 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1730358 T5995 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 1730361 T5969 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
>   [junit4]   2> 1730367 T5995 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 1730369 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1730372 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1730373 T5979 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 1730367 T5954 C4181 P53698 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=http://127.0.0.1:53698/od_jjq/cq DONE.  We have no versions.  sync failed.
>   [junit4]   2> 1730374 T5954 C4181 P53698 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
>   [junit4]   2> 1730374 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
>   [junit4]   2> 1730379 T5946 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1730379 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53698/od_jjq/cq/solrj_collection_shard1_replica1/ shard1
>   [junit4]   2> 1730375 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 1730384 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1730382 T6001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1730381 T5973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1730380 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1730385 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
>   [junit4]   2> 1730384 T5954 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard1
>   [junit4]   2> 1730391 T5969 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=-9
>   [junit4]   2> 1730395 T5969 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
>   [junit4]   2> 1730396 T5969 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> 1730396 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 1730397 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 1730397 T5979 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 1730395 T5995 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 1730399 T5995 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 1730398 T5979 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/wo
> 
> [...truncated too long message...]
> 
> )
>   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
>   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 	
>   [junit4]   2> 2162194 T5987 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/nodes_used_collection/leader_elect/shard1/election/91100191848005644-core_node1-n_0000000001
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
>   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
>   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
>   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 
>   [junit4]   2> 2162195 T5987 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>   [junit4]   2> 2162195 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
>   [junit4]   2> 2162195 T5987 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>   [junit4]   2> 2162196 T5987 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 
>   [junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_IQ, timezone=Etc/UTC
>   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=1,free=107685104,total=385253376
>   [junit4]   2> NOTE: All tests run in this JVM: [PluginInfoTest, QueryElevationComponentTest, ZkCLITest, TestCollationField, CurrencyFieldXmlFileTest, BadCopyFieldTest, TestSolrXmlPersistence, TestPartialUpdateDeduplication, TestManagedSchema, WordBreakSolrSpellCheckerTest, RAMDirectoryFactoryTest, TestDistributedMissingSort, TestJmxIntegration, TestElisionMultitermQuery, SpellCheckComponentTest, ZkSolrClientTest, HdfsBasicDistributedZk2Test, TestArbitraryIndexDir, TestSimpleQParserPlugin, TestRandomDVFaceting, TestSchemaSimilarityResource, OutputWriterTest, TestQueryTypes, CopyFieldTest, TestStressReorder, DistanceFunctionTest, SuggestComponentTest, MigrateRouteKeyTest, DateFieldTest, DistributedDebugComponentTest, TestCloudManagedSchemaAddField, FieldAnalysisRequestHandlerTest, TestFileDictionaryLookup, TestDocBasedVersionConstraints, LeaderElectionIntegrationTest, TestCollapseQParserPlugin, TestMultiCoreConfBootstrap, QueryResultKeyTest, TestDocSet, TestLMDirichletSimilarityFactory, TestCSVLoader, TestRTGBase, TestSchemaVersionResource, FileUtilsTest, TriLevelCompositeIdRoutingTest, ParsingFieldUpdateProcessorsTest, TestCodecSupport, DistributedTermsComponentTest, TestCustomSort, ConnectionManagerTest, RequiredFieldsTest, TestQueryUtils, TestReplicationHandler, TestSolrXMLSerializer, TestUtils, UpdateRequestProcessorFactoryTest, TestFastWriter, TestPHPSerializedResponseWriter, OpenExchangeRatesOrgProviderTest, TestFoldingMultitermQuery, TestNonNRTOpen, DOMUtilTest, CSVRequestHandlerTest, TestIndexingPerformance, DocumentAnalysisRequestHandlerTest, TestLuceneMatchVersion, DocValuesTest, TestJoin, TestSchemaNameResource, TestBadConfig, CoreAdminCreateDiscoverTest, HighlighterConfigTest, TestPerFieldSimilarity, BlockCacheTest, SolrIndexSplitterTest, URLClassifyProcessorTest, SchemaVersionSpecificBehaviorTest, QueryParsingTest, CursorMarkTest, SimplePostToolTest, TestNumberUtils, EnumFieldTest, ChaosMonkeySafeLeaderTest, DirectUpdateHandlerOptimizeTest, TestCollationFieldDocValues, TestFuzzyAnalyzedSuggestions, TestQuerySenderListener, HdfsRecoveryZkTest, CoreMergeIndexesAdminHandlerTest, ClusterStateTest, TestSolr4Spatial, TestSystemIdResolver, TestStressLucene, SolrCoreCheckLockOnStartupTest, TestManagedSchemaFieldResource, IndexSchemaTest, TestWordDelimiterFilterFactory, TestOmitPositions, TermVectorComponentTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestIndexSearcher, TestPseudoReturnFields, HdfsChaosMonkeySafeLeaderTest, TestLazyCores, DateMathParserTest, TestAtomicUpdateErrorCases, TestLFUCache, TestDefaultSimilarityFactory, TestPostingsSolrHighlighter, SimpleFacetsTest, TestImplicitCoreProperties, DeleteReplicaTest, TestSerializedLuceneMatchVersion, TestHighFrequencyDictionaryFactory, ChangedSchemaMergeTest, TestAnalyzedSuggestions, CursorPagingTest, PingRequestHandlerTest, TestSuggestSpellingConverter, TestDFRSimilarityFactory, TestSolrDeletionPolicy2, PrimUtilsTest, SignatureUpdateProcessorFactoryTest, TestStressUserVersions, TestDistribDocBasedVersion, PrimitiveFieldTypeTest, ShardRoutingCustomTest, TestCloudManagedSchema, TestDynamicFieldCollectionResource, AutoCommitTest, SolrPluginUtilsTest, BlockDirectoryTest, SolrIndexConfigTest, ShowFileRequestHandlerTest, JSONWriterTest, TestStressVersions, SpatialFilterTest, BadIndexSchemaTest, ShardRoutingTest, TestShardHandlerFactory, SuggesterTest, TestBM25SimilarityFactory, DirectUpdateHandlerTest, DistribCursorPagingTest, FieldMutatingUpdateProcessorTest, RecoveryZkTest, TestNoOpRegenerator, TestSolrDeletionPolicy1, BasicDistributedZkTest, TestBinaryResponseWriter, TestRandomMergePolicy, ExternalFileFieldSortTest, OpenCloseCoreStressTest, PolyFieldTest, BasicDistributedZk2Test, TestFreeTextSuggestions, JsonLoaderTest, TestAddFieldRealTimeGet, UniqFieldsUpdateProcessorFactoryTest, TestAnalyzeInfixSuggestions, TestStressRecovery, TestBinaryField, IndexSchemaRuntimeFieldTest, TestSolrQueryParser, HdfsBasicDistributedZkTest, ZkNodePropsTest, BadComponentTest, UUIDFieldTest, AssignTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
>   [junit4] Completed in 443.02s, 1 test, 1 failure <<< FAILURES!
> 
> [...truncated 539 lines...]
> BUILD FAILED
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1540 tests, 1 failure, 40 ignored (6 assumptions)
> 
> Total time: 85 minutes 49 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1210 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1210/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
	at __randomizedtesting.SeedInfo.seed([E01973DE033FE27B:61FFFDC674608247]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:772)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:196)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10429 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 1719213 T5915 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /od_jjq/cq
   [junit4]   2> 1719222 T5915 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1390078610952
   [junit4]   2> 1719224 T5915 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1719225 T5916 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1719327 T5915 oasc.ZkTestServer.run start zk server on port:53689
   [junit4]   2> 1719329 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1719338 T5922 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b2bf8c1 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1719338 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1719339 T5915 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1719353 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1719358 T5924 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31e8fc2a name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1719359 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1719359 T5915 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1719375 T5915 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1719384 T5915 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1719393 T5915 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1719403 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1719404 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1719421 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1719422 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1719432 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1719433 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1719444 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1719445 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1719456 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1719457 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1719469 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1719470 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1719480 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1719481 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1719492 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1719493 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1719503 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1719504 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1719513 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1719514 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1719524 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1719526 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 1719542 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1719547 T5926 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66930f41 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1719547 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1719548 T5915 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1719558 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1719562 T5928 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3239c4d9 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1719562 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1719564 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 1719565 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 1719576 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 1719578 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 1719588 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1719589 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1719600 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 1719600 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 1719611 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 1719612 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 1719622 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 1719624 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 1719633 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1719634 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1719650 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1719651 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1719683 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 1719684 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 1719694 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 1719695 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 1719706 T5915 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 1719707 T5915 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 1720156 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1720161 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53694
   [junit4]   2> 1720162 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1720163 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1720164 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458
   [junit4]   2> 1720164 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458/'
   [junit4]   2> 1720208 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458/solr.xml
   [junit4]   2> 1720327 T5915 oasc.CoreContainer.<init> New CoreContainer 370136241
   [junit4]   2> 1720328 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1390078611458/]
   [junit4]   2> 1720330 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1720330 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1720331 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1720332 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1720333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1720333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1720334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1720334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1720335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1720350 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1720351 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1720352 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1720352 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
   [junit4]   2> 1720353 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1720355 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1720360 T5939 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6954bbc7 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1720361 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1720370 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1720374 T5941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f18b9ff name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1720375 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1720381 T5915 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1720394 T5915 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1720406 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1720412 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53694_od_jjq%2Fcq
   [junit4]   2> 1720416 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53694_od_jjq%2Fcq
   [junit4]   2> 1720428 T5915 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1720436 T5915 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1720451 T5915 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1720460 T5915 oasc.Overseer.start Overseer (id=91100191848005637-127.0.0.1:53694_od_jjq%2Fcq-n_0000000000) starting
   [junit4]   2> 1720471 T5915 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1720486 T5943 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1720488 T5915 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1720490 T5943 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1720497 T5915 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1720504 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1720515 T5942 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1720523 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1720524 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1720527 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1720532 T5946 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fdc9238 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1720532 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1720536 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1720546 T5915 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1721004 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1721012 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53698
   [junit4]   2> 1721013 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1721013 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1721014 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275
   [junit4]   2> 1721014 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/'
   [junit4]   2> 1721059 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solr.xml
   [junit4]   2> 1721165 T5915 oasc.CoreContainer.<init> New CoreContainer 2108103389
   [junit4]   2> 1721165 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/]
   [junit4]   2> 1721167 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1721167 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1721167 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1721168 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1721168 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1721168 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1721169 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1721169 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1721169 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1721186 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1721187 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1721187 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1721188 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
   [junit4]   2> 1721189 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1721192 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1721207 T5957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65d4e39d name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1721207 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1721220 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1721228 T5959 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4db4e5c6 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1721229 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1721243 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1722255 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53698_od_jjq%2Fcq
   [junit4]   2> 1722258 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53698_od_jjq%2Fcq
   [junit4]   2> 1722266 T5959 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1722266 T5941 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1722267 T5946 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1722281 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1722281 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1722625 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1722630 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53701
   [junit4]   2> 1722631 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1722632 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1722632 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011
   [junit4]   2> 1722632 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/'
   [junit4]   2> 1722668 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solr.xml
   [junit4]   2> 1722738 T5915 oasc.CoreContainer.<init> New CoreContainer 1501927636
   [junit4]   2> 1722739 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/]
   [junit4]   2> 1722741 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1722741 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1722741 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1722742 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1722743 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1722743 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1722753 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1722753 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1722753 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1722754 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
   [junit4]   2> 1722754 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1722755 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1722760 T5971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51f4b436 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1722760 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1722767 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1722771 T5973 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b11c2d8 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1722771 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1722782 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1723792 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53701_od_jjq%2Fcq
   [junit4]   2> 1723795 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53701_od_jjq%2Fcq
   [junit4]   2> 1723805 T5941 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1723806 T5946 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1723807 T5959 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1723807 T5973 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1723827 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1723828 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1724210 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1724215 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53704
   [junit4]   2> 1724216 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1724216 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1724217 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557
   [junit4]   2> 1724217 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/'
   [junit4]   2> 1724246 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solr.xml
   [junit4]   2> 1724331 T5915 oasc.CoreContainer.<init> New CoreContainer 1311775510
   [junit4]   2> 1724332 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/]
   [junit4]   2> 1724333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1724333 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1724334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1724334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1724334 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1724335 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1724346 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1724347 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1724347 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1724347 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
   [junit4]   2> 1724348 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1724349 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1724354 T5985 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38426974 name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1724354 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1724361 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1724364 T5987 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22116710 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1724365 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1724377 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1725388 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53704_od_jjq%2Fcq
   [junit4]   2> 1725391 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53704_od_jjq%2Fcq
   [junit4]   2> 1725402 T5946 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1725402 T5941 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1725403 T5959 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1725403 T5987 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1725403 T5973 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1725420 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1725421 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1725880 T5915 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1725885 T5915 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53707
   [junit4]   2> 1725886 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1725886 T5915 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1725887 T5915 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150
   [junit4]   2> 1725888 T5915 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/'
   [junit4]   2> 1725922 T5915 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solr.xml
   [junit4]   2> 1725993 T5915 oasc.CoreContainer.<init> New CoreContainer 1201695941
   [junit4]   2> 1725994 T5915 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/]
   [junit4]   2> 1725995 T5915 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1725995 T5915 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1725996 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1725997 T5915 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1725997 T5915 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1725998 T5915 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1726008 T5915 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1726008 T5915 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1726008 T5915 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1726009 T5915 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53689/solr
   [junit4]   2> 1726009 T5915 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1726010 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1726014 T5999 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75451e3f name:ZooKeeperConnection Watcher:127.0.0.1:53689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1726015 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1726022 T5915 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1726028 T6001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@509b80a4 name:ZooKeeperConnection Watcher:127.0.0.1:53689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1726028 T5915 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1726040 T5915 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1727054 T5915 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53707_od_jjq%2Fcq
   [junit4]   2> 1727056 T5915 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53707_od_jjq%2Fcq
   [junit4]   2> 1727068 T5959 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1727068 T5946 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1727068 T5987 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1727068 T6001 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1727068 T5941 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1727068 T5973 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1727088 T5915 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1727088 T5915 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1727135 T5953 oasha.CollectionsHandler.handleCreateAction Creating Collection : version=2&numShards=2&action=CREATE&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&replicationFactor=2&wt=javabin&router.name=compositeId
   [junit4]   2> 1727138 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1727142 T5943 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 1727142 T5943 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 1727143 T5943 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1727143 T5943 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 1727156 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1727160 T5942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1727161 T5942 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 1727162 T5942 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 1727169 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1727172 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727172 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727172 T5946 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727172 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727172 T5973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727172 T6001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727257 T5943 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 1727258 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:53698_od_jjq%2Fcq
   [junit4]   2> 1727259 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:53707_od_jjq%2Fcq
   [junit4]   2> 1727260 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:53704_od_jjq%2Fcq
   [junit4]   2> 1727262 T5943 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:53701_od_jjq%2Fcq
   [junit4]   2> 1727267 T5954 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 1727271 T5979 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 1727273 T5995 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 1727273 T5969 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 1727276 T5954 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1727277 T5979 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1727277 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1727279 T5969 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1727281 T5995 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1727284 T5942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1727287 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53698/od_jjq/cq",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53698_od_jjq%2Fcq",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1727295 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1727300 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53704/od_jjq/cq",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53704_od_jjq%2Fcq",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1727310 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1727314 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53701/od_jjq/cq",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53701_od_jjq%2Fcq",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1727321 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1727324 T5942 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53707/od_jjq/cq",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53707_od_jjq%2Fcq",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1727332 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1727445 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727445 T5973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727446 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727445 T6001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727446 T5946 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1727446 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1728278 T5954 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1728278 T5979 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1728279 T5954 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solr.xml
   [junit4]   2> 1728279 T5979 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solr.xml
   [junit4]   2> 1728280 T5979 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solrj_collection_shard2_replica1
   [junit4]   2> 1728280 T5979 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1728281 T5969 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1728280 T5954 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1
   [junit4]   2> 1728281 T5954 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1728282 T5969 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solr.xml
   [junit4]   2> 1728282 T5969 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2
   [junit4]   2> 1728282 T5969 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1728282 T5995 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1728284 T5995 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solr.xml
   [junit4]   2> 1728284 T5979 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1728284 T5995 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2
   [junit4]   2> 1728284 T5954 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1728285 T5954 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1728285 T5995 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1728285 T5969 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1728286 T5969 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1728284 T5979 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1728287 T5995 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1728287 T5995 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1728289 T5954 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/'
   [junit4]   2> 1728291 T5995 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/'
   [junit4]   2> 1728292 T5969 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/'
   [junit4]   2> 1728295 T5979 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1390078615557/solrj_collection_shard2_replica1/'
   [junit4]   2> 1728399 T5954 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1728413 T5995 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1728426 T5969 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1728492 T5979 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1728547 T5969 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1728550 T5995 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1728554 T5969 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1728574 T5995 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1728577 T5969 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 1728610 T5995 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 1728640 T5954 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1728652 T5979 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1728653 T5954 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1728663 T5954 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 1728663 T5979 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1728715 T5979 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 1730003 T5954 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1730003 T5954 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1730004 T5954 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1730012 T5954 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1730018 T5954 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1730071 T5969 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1730071 T5969 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1730073 T5969 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1730090 T5969 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1730097 T5969 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1730128 T5995 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1730128 T5995 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1730129 T5995 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1730129 T5954 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1730137 T5969 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1730141 T5954 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1730153 T5995 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1730158 T5995 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1730166 T5969 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1730176 T5954 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1730180 T5969 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1730181 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1730182 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1730182 T5954 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1730182 T5979 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1730184 T5979 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1730184 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1730184 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1730185 T5969 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1730186 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1730186 T5954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1730187 T5954 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1730187 T5954 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/
   [junit4]   2> 1730188 T5954 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@589e0cdc
   [junit4]   2> 1730185 T5979 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1730189 T5954 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data
   [junit4]   2> 1730189 T5954 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 1730190 T5954 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1730190 T5954 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1390078612275/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 1730191 T5954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1730192 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1730192 T5969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1730192 T5969 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1730193 T5969 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/
   [junit4]   2> 1730193 T5969 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@589e0cdc
   [junit4]   2> 1730194 T5969 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data
   [junit4]   2> 1730195 T5969 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 1730195 T5969 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1730197 T5969 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1390078614011/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 1730197 T5969 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1730199 T5954 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@247327d5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@392d8b48),segFN=segments_1,generation=1}
   [junit4]   2> 1730199 T5954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1730199 T5969 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@434ea2f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4d5719cb),segFN=segments_1,generation=1}
   [junit4]   2> 1730200 T5969 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1730200 T5979 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1730204 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1730204 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1730205 T5969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1730206 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1730207 T5969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1730207 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1730208 T5969 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1730209 T5969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1730209 T5969 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1730204 T5979 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1730210 T5969 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1730212 T5969 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1730213 T5969 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1730213 T5969 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1730214 T5969 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1730215 T5969 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1730209 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1730216 T5969 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1730216 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1730217 T5954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1730217 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1730218 T5954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1730219 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1730220 T5954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1730220 T5954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1730221 T5954 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1730222 T5954 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1730222 T5954 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1730223 T5954 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1730223 T5954 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1730224 T5954 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1730227 T5954 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1730226 T5969 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1730228 T5954 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1730235 T5954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1730240 T5969 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1730241 T5969 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1730244 T5969 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=43, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6018786168657202]
   [junit4]   2> 1730245 T5969 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@434ea2f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4d5719cb),segFN=segments_1,generation=1}
   [junit4]   2> 1730245 T5969 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1730245 T5969 oass.SolrIndexSearcher.<init> Opening Searcher@682747bb main
   [junit4]   2> 1730246 T5954 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1730249 T5954 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1730249 T5995 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1730251 T5954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=43, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6018786168657202]
   [junit4]   2> 1730253 T5954 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@247327d5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@392d8b48),segFN=segments_1,generation=1}
   [junit4]   2> 1730253 T5954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1730254 T5954 oass.SolrIndexSearcher.<init> Opening Searcher@7487fd38 main
   [junit4]   2> 1730260 T6008 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@7487fd38 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1730263 T5995 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1730263 T5954 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
   [junit4]   2> 1730264 T5954 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:http://127.0.0.1:53698/od_jjq/cq collection:solrj_collection shard:shard1
   [junit4]   2> 1730264 T6007 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@682747bb main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1730268 T5954 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 1730288 T5995 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1730288 T5969 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
   [junit4]   2> 1730289 T5969 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica2 address:http://127.0.0.1:53701/od_jjq/cq collection:solrj_collection shard:shard2
   [junit4]   2> 1730292 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1730293 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1730294 T5995 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1730300 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1730301 T5995 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1730302 T5995 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1730302 T5995 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/
   [junit4]   2> 1730303 T5995 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@589e0cdc
   [junit4]   2> 1730294 T5969 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 1730308 T5995 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data
   [junit4]   2> 1730309 T5995 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 1730310 T5995 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1730311 T5995 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390078617150/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 1730312 T5995 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1730319 T5995 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@858afaf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@659dfd2a),segFN=segments_1,generation=1}
   [junit4]   2> 1730320 T5995 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1730322 T5979 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1730325 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1730331 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1730332 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1730333 T5995 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1730333 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1730334 T5995 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1730335 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1730336 T5995 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1730338 T5995 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1730339 T5995 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1730339 T5979 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1730340 T5995 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1730341 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1730343 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1730343 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1730344 T5995 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1730344 T5995 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1730346 T5995 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1730346 T5954 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=-4
   [junit4]   2> 1730349 T5954 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
   [junit4]   2> 1730349 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4181 name=solrj_collection_shard1_replica1 org.apache.solr.core.SolrCore@46d7249d url=http://127.0.0.1:53698/od_jjq/cq/solrj_collection_shard1_replica1 node=127.0.0.1:53698_od_jjq%2Fcq C4181_STATE=coll:solrj_collection core:solrj_collection_shard1_replica1 props:{state=down, base_url=http://127.0.0.1:53698/od_jjq/cq, core=solrj_collection_shard1_replica1, node_name=127.0.0.1:53698_od_jjq%2Fcq}
   [junit4]   2> 1730350 T5954 C4181 P53698 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53698/od_jjq/cq/solrj_collection_shard1_replica1/
   [junit4]   2> 1730350 T5954 C4181 P53698 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=http://127.0.0.1:53698/od_jjq/cq START replicas=[http://127.0.0.1:53707/od_jjq/cq/solrj_collection_shard1_replica2/] nUpdates=100
   [junit4]   2> 1730353 T5995 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1730354 T5995 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1730356 T5942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1730354 T5979 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1730358 T5995 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1730361 T5969 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1730367 T5995 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1730369 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1730372 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1730373 T5979 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1730367 T5954 C4181 P53698 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=http://127.0.0.1:53698/od_jjq/cq DONE.  We have no versions.  sync failed.
   [junit4]   2> 1730374 T5954 C4181 P53698 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1730374 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1730379 T5946 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1730379 T5954 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53698/od_jjq/cq/solrj_collection_shard1_replica1/ shard1
   [junit4]   2> 1730375 T5941 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1730384 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1730382 T6001 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1730381 T5973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1730380 T5959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1730385 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1730384 T5954 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard1
   [junit4]   2> 1730391 T5969 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=-9
   [junit4]   2> 1730395 T5969 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
   [junit4]   2> 1730396 T5969 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1730396 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1730397 T5979 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1730397 T5979 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1730395 T5995 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1730399 T5995 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1730398 T5979 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/wo

[...truncated too long message...]

)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 2162194 T5987 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/nodes_used_collection/leader_elect/shard1/election/91100191848005644-core_node1-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2162195 T5987 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2162195 T5987 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2162195 T5987 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2162196 T5987 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_IQ, timezone=Etc/UTC
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=1,free=107685104,total=385253376
   [junit4]   2> NOTE: All tests run in this JVM: [PluginInfoTest, QueryElevationComponentTest, ZkCLITest, TestCollationField, CurrencyFieldXmlFileTest, BadCopyFieldTest, TestSolrXmlPersistence, TestPartialUpdateDeduplication, TestManagedSchema, WordBreakSolrSpellCheckerTest, RAMDirectoryFactoryTest, TestDistributedMissingSort, TestJmxIntegration, TestElisionMultitermQuery, SpellCheckComponentTest, ZkSolrClientTest, HdfsBasicDistributedZk2Test, TestArbitraryIndexDir, TestSimpleQParserPlugin, TestRandomDVFaceting, TestSchemaSimilarityResource, OutputWriterTest, TestQueryTypes, CopyFieldTest, TestStressReorder, DistanceFunctionTest, SuggestComponentTest, MigrateRouteKeyTest, DateFieldTest, DistributedDebugComponentTest, TestCloudManagedSchemaAddField, FieldAnalysisRequestHandlerTest, TestFileDictionaryLookup, TestDocBasedVersionConstraints, LeaderElectionIntegrationTest, TestCollapseQParserPlugin, TestMultiCoreConfBootstrap, QueryResultKeyTest, TestDocSet, TestLMDirichletSimilarityFactory, TestCSVLoader, TestRTGBase, TestSchemaVersionResource, FileUtilsTest, TriLevelCompositeIdRoutingTest, ParsingFieldUpdateProcessorsTest, TestCodecSupport, DistributedTermsComponentTest, TestCustomSort, ConnectionManagerTest, RequiredFieldsTest, TestQueryUtils, TestReplicationHandler, TestSolrXMLSerializer, TestUtils, UpdateRequestProcessorFactoryTest, TestFastWriter, TestPHPSerializedResponseWriter, OpenExchangeRatesOrgProviderTest, TestFoldingMultitermQuery, TestNonNRTOpen, DOMUtilTest, CSVRequestHandlerTest, TestIndexingPerformance, DocumentAnalysisRequestHandlerTest, TestLuceneMatchVersion, DocValuesTest, TestJoin, TestSchemaNameResource, TestBadConfig, CoreAdminCreateDiscoverTest, HighlighterConfigTest, TestPerFieldSimilarity, BlockCacheTest, SolrIndexSplitterTest, URLClassifyProcessorTest, SchemaVersionSpecificBehaviorTest, QueryParsingTest, CursorMarkTest, SimplePostToolTest, TestNumberUtils, EnumFieldTest, ChaosMonkeySafeLeaderTest, DirectUpdateHandlerOptimizeTest, TestCollationFieldDocValues, TestFuzzyAnalyzedSuggestions, TestQuerySenderListener, HdfsRecoveryZkTest, CoreMergeIndexesAdminHandlerTest, ClusterStateTest, TestSolr4Spatial, TestSystemIdResolver, TestStressLucene, SolrCoreCheckLockOnStartupTest, TestManagedSchemaFieldResource, IndexSchemaTest, TestWordDelimiterFilterFactory, TestOmitPositions, TermVectorComponentTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestIndexSearcher, TestPseudoReturnFields, HdfsChaosMonkeySafeLeaderTest, TestLazyCores, DateMathParserTest, TestAtomicUpdateErrorCases, TestLFUCache, TestDefaultSimilarityFactory, TestPostingsSolrHighlighter, SimpleFacetsTest, TestImplicitCoreProperties, DeleteReplicaTest, TestSerializedLuceneMatchVersion, TestHighFrequencyDictionaryFactory, ChangedSchemaMergeTest, TestAnalyzedSuggestions, CursorPagingTest, PingRequestHandlerTest, TestSuggestSpellingConverter, TestDFRSimilarityFactory, TestSolrDeletionPolicy2, PrimUtilsTest, SignatureUpdateProcessorFactoryTest, TestStressUserVersions, TestDistribDocBasedVersion, PrimitiveFieldTypeTest, ShardRoutingCustomTest, TestCloudManagedSchema, TestDynamicFieldCollectionResource, AutoCommitTest, SolrPluginUtilsTest, BlockDirectoryTest, SolrIndexConfigTest, ShowFileRequestHandlerTest, JSONWriterTest, TestStressVersions, SpatialFilterTest, BadIndexSchemaTest, ShardRoutingTest, TestShardHandlerFactory, SuggesterTest, TestBM25SimilarityFactory, DirectUpdateHandlerTest, DistribCursorPagingTest, FieldMutatingUpdateProcessorTest, RecoveryZkTest, TestNoOpRegenerator, TestSolrDeletionPolicy1, BasicDistributedZkTest, TestBinaryResponseWriter, TestRandomMergePolicy, ExternalFileFieldSortTest, OpenCloseCoreStressTest, PolyFieldTest, BasicDistributedZk2Test, TestFreeTextSuggestions, JsonLoaderTest, TestAddFieldRealTimeGet, UniqFieldsUpdateProcessorFactoryTest, TestAnalyzeInfixSuggestions, TestStressRecovery, TestBinaryField, IndexSchemaRuntimeFieldTest, TestSolrQueryParser, HdfsBasicDistributedZkTest, ZkNodePropsTest, BadComponentTest, UUIDFieldTest, AssignTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed in 443.02s, 1 test, 1 failure <<< FAILURES!

[...truncated 539 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1540 tests, 1 failure, 40 ignored (6 assumptions)

Total time: 85 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.6.0) - Build # 1209 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1209/
Java: 64bit/jdk1.6.0 -XX:-UseCompressedOops -XX:+UseParallelGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
	at __randomizedtesting.SeedInfo.seed([8481D970B6D56594:5675768C18A05A8]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:772)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:196)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:695)


FAILED:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
could not set the new overseer

Stack Trace:
java.lang.AssertionError: could not set the new overseer
	at __randomizedtesting.SeedInfo.seed([8481D970B6D56594:5675768C18A05A8]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.OverseerRolesTest.addOverseerRole2ExistingNodes(OverseerRolesTest.java:120)
	at org.apache.solr.cloud.OverseerRolesTest.doTest(OverseerRolesTest.java:86)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:695)




Build Log:
[...truncated 9359 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 417551 T1395 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 417559 T1395 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1390052631591
   [junit4]   2> 417561 T1395 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 417562 T1396 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 417664 T1395 oasc.ZkTestServer.run start zk server on port:52595
   [junit4]   2> 417665 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 417675 T1402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38f8cf74 name:ZooKeeperConnection Watcher:127.0.0.1:52595 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 417675 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 417676 T1395 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 417706 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 417724 T1404 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9349448 name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 417724 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 417724 T1395 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 417735 T1395 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 417746 T1395 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 417754 T1395 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 417764 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 417765 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 417779 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 417780 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 417791 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 417791 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 417801 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 417802 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 417814 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 417814 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 417824 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 417825 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 417835 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 417835 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 417845 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 417845 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 417855 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 417855 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 417865 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 417866 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 417876 T1395 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 417877 T1395 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 418257 T1395 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 418268 T1395 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52598
   [junit4]   2> 418269 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 418269 T1395 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 418270 T1395 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1390052631922
   [junit4]   2> 418270 T1395 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1390052631922/'
   [junit4]   2> 418296 T1395 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1390052631922/solr.xml
   [junit4]   2> 418339 T1395 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 418340 T1395 oasc.CoreContainer.<init> New CoreContainer 1008860487
   [junit4]   2> 418340 T1395 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1390052631922/]
   [junit4]   2> 418342 T1395 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 418343 T1395 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 418343 T1395 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 418344 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 418344 T1395 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 418344 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 418345 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 418345 T1395 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 418346 T1395 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 418370 T1395 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 418371 T1395 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 418371 T1395 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 418372 T1395 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52595/solr
   [junit4]   2> 418372 T1395 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 418374 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 418379 T1415 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b7b5f28 name:ZooKeeperConnection Watcher:127.0.0.1:52595 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 418380 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 418389 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 418394 T1417 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61285ffb name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 418395 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 418401 T1395 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 418415 T1395 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 418429 T1395 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 418437 T1395 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52598_
   [junit4]   2> 418440 T1395 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52598_
   [junit4]   2> 418454 T1395 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 418464 T1395 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 418480 T1395 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 418498 T1395 oasc.Overseer.start Overseer (id=91098489264406531-127.0.0.1:52598_-n_0000000000) starting
   [junit4]   2> 418515 T1395 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 418534 T1419 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 418536 T1395 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 418537 T1419 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 418555 T1395 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 418563 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 418578 T1418 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 418583 T1395 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1390052631922
   [junit4]   2> 418688 T1395 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1390052631922/conf/
   [junit4]   2> 418692 T1395 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 418693 T1395 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 418694 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 418695 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 418701 T1422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d4b3266 name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 418701 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 418706 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 418717 T1395 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 419042 T1395 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 419048 T1395 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52602
   [junit4]   2> 419049 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 419050 T1395 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 419050 T1395 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1390052632747
   [junit4]   2> 419050 T1395 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1390052632747/'
   [junit4]   2> 419078 T1395 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1390052632747/solr.xml
   [junit4]   2> 419125 T1395 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 419125 T1395 oasc.CoreContainer.<init> New CoreContainer 2101251385
   [junit4]   2> 419126 T1395 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1390052632747/]
   [junit4]   2> 419127 T1395 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 419127 T1395 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 419128 T1395 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 419128 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 419128 T1395 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 419129 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 419129 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 419129 T1395 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 419130 T1395 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 419157 T1395 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 419157 T1395 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 419159 T1395 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 419159 T1395 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52595/solr
   [junit4]   2> 419159 T1395 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 419162 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 419167 T1433 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15a9d96c name:ZooKeeperConnection Watcher:127.0.0.1:52595 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 419168 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 419182 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 419184 T1435 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cebbff7 name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 419184 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 419202 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 420216 T1395 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52602_
   [junit4]   2> 420219 T1395 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52602_
   [junit4]   2> 420234 T1435 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 420235 T1422 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 420241 T1417 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 420256 T1395 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1390052632747
   [junit4]   2> 420397 T1395 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1390052632747/conf/
   [junit4]   2> 420404 T1395 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 420405 T1395 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 420405 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 420691 T1395 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 420696 T1395 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52605
   [junit4]   2> 420698 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 420698 T1395 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 420699 T1395 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1390052634436
   [junit4]   2> 420699 T1395 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1390052634436/'
   [junit4]   2> 420725 T1395 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1390052634436/solr.xml
   [junit4]   2> 420764 T1395 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 420765 T1395 oasc.CoreContainer.<init> New CoreContainer 1190029155
   [junit4]   2> 420765 T1395 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1390052634436/]
   [junit4]   2> 420766 T1395 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 420767 T1395 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 420767 T1395 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 420768 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 420768 T1395 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 420768 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 420769 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 420769 T1395 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 420769 T1395 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 420791 T1395 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 420792 T1395 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 420792 T1395 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 420792 T1395 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52595/solr
   [junit4]   2> 420793 T1395 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 420795 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 420801 T1447 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4dc5b8f6 name:ZooKeeperConnection Watcher:127.0.0.1:52595 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 420802 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 420810 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 420815 T1449 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ba5f3e5 name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 420815 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 420830 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 421843 T1395 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52605_
   [junit4]   2> 421847 T1395 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52605_
   [junit4]   2> 421864 T1435 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 421865 T1449 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 421867 T1422 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 421870 T1417 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 421890 T1395 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1390052634436
   [junit4]   2> 422017 T1395 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1390052634436/conf/
   [junit4]   2> 422023 T1395 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 422024 T1395 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 422024 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 422395 T1395 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 422401 T1395 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52608
   [junit4]   2> 422403 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 422404 T1395 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 422404 T1395 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1390052636055
   [junit4]   2> 422405 T1395 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1390052636055/'
   [junit4]   2> 422434 T1395 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1390052636055/solr.xml
   [junit4]   2> 422478 T1395 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 422478 T1395 oasc.CoreContainer.<init> New CoreContainer 1376667533
   [junit4]   2> 422479 T1395 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty3-1390052636055/]
   [junit4]   2> 422480 T1395 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 422480 T1395 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 422481 T1395 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 422481 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 422482 T1395 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 422482 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 422482 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 422483 T1395 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 422483 T1395 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 422510 T1395 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 422511 T1395 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 422512 T1395 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 422512 T1395 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52595/solr
   [junit4]   2> 422512 T1395 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 422514 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 422518 T1461 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@277ba411 name:ZooKeeperConnection Watcher:127.0.0.1:52595 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 422519 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 422527 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 422532 T1463 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@446dcf38 name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 422532 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 422546 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 423561 T1395 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52608_
   [junit4]   2> 423564 T1395 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52608_
   [junit4]   2> 423578 T1417 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 423580 T1435 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 423581 T1422 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 423581 T1449 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 423583 T1463 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 423604 T1395 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1390052636055
   [junit4]   2> 423712 T1395 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1390052636055/conf/
   [junit4]   2> 423719 T1395 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 423720 T1395 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 423721 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 424036 T1395 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 424041 T1395 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52611
   [junit4]   2> 424044 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 424045 T1395 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 424045 T1395 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752
   [junit4]   2> 424045 T1395 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752/'
   [junit4]   2> 424070 T1395 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752/solr.xml
   [junit4]   2> 424131 T1395 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 424132 T1395 oasc.CoreContainer.<init> New CoreContainer 349731220
   [junit4]   2> 424133 T1395 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752/]
   [junit4]   2> 424134 T1395 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 424135 T1395 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 424135 T1395 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 424135 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 424136 T1395 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 424136 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 424136 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 424137 T1395 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 424137 T1395 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 424160 T1395 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 424161 T1395 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 424161 T1395 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 424161 T1395 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52595/solr
   [junit4]   2> 424162 T1395 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 424163 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 424170 T1475 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1015ee3d name:ZooKeeperConnection Watcher:127.0.0.1:52595 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 424170 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 424179 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 424183 T1477 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@694ed89a name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 424184 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 424202 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 425216 T1395 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52611_
   [junit4]   2> 425219 T1395 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52611_
   [junit4]   2> 425232 T1449 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 425237 T1422 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 425238 T1435 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 425238 T1463 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 425239 T1417 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 425242 T1477 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 425259 T1395 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752
   [junit4]   2> 425409 T1395 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752/conf/
   [junit4]   2> 425418 T1395 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 425419 T1395 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 425419 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 425717 T1395 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 425722 T1395 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52614
   [junit4]   2> 425723 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 425724 T1395 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 425724 T1395 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1390052639449
   [junit4]   2> 425724 T1395 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1390052639449/'
   [junit4]   2> 425749 T1395 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1390052639449/solr.xml
   [junit4]   2> 425788 T1395 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 425789 T1395 oasc.CoreContainer.<init> New CoreContainer 564885335
   [junit4]   2> 425789 T1395 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1390052639449/]
   [junit4]   2> 425790 T1395 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 425791 T1395 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 425791 T1395 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 425791 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 425791 T1395 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 425792 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 425792 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 425792 T1395 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 425793 T1395 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 425814 T1395 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 425815 T1395 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 425816 T1395 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 425816 T1395 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52595/solr
   [junit4]   2> 425816 T1395 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 425818 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 425823 T1489 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@280206 name:ZooKeeperConnection Watcher:127.0.0.1:52595 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 425823 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 425835 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 425839 T1491 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74bedbdd name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 425840 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 425855 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 426871 T1395 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52614_
   [junit4]   2> 426875 T1395 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52614_
   [junit4]   2> 426899 T1417 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 426900 T1449 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 426901 T1477 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 426903 T1435 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 426904 T1422 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 426904 T1463 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 426905 T1491 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 426926 T1395 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1390052639449
   [junit4]   2> 427020 T1395 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1390052639449/conf/
   [junit4]   2> 427025 T1395 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 427026 T1395 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 427026 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 427386 T1395 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 427390 T1395 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52617
   [junit4]   2> 427396 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 427396 T1395 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 427397 T1395 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1390052641057
   [junit4]   2> 427397 T1395 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1390052641057/'
   [junit4]   2> 427437 T1395 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1390052641057/solr.xml
   [junit4]   2> 427499 T1395 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 427500 T1395 oasc.CoreContainer.<init> New CoreContainer 169730854
   [junit4]   2> 427500 T1395 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1390052641057/]
   [junit4]   2> 427502 T1395 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 427502 T1395 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 427503 T1395 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 427504 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 427504 T1395 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 427504 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 427505 T1395 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 427505 T1395 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 427506 T1395 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 427543 T1395 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 427544 T1395 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 427544 T1395 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 427545 T1395 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52595/solr
   [junit4]   2> 427546 T1395 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 427548 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 427554 T1503 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@302bd8fa name:ZooKeeperConnection Watcher:127.0.0.1:52595 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 427555 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 427566 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 427571 T1505 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b870011 name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 427572 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 427591 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 428612 T1395 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52617_
   [junit4]   2> 428616 T1395 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52617_
   [junit4]   2> 428632 T1463 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 428636 T1422 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 428638 T1435 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 428639 T1417 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 428641 T1449 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 428642 T1477 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 428643 T1491 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 428646 T1505 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 428668 T1395 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1390052641057
   [junit4]   2> 428811 T1395 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1390052641057/conf/
   [junit4]   2> 428819 T1395 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 428820 T1395 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 428821 T1395 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 428823 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 428828 T1508 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39203412 name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 428829 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 428836 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 428848 T1395 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 428853 T1510 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e498ef2 name:ZooKeeperConnection Watcher:127.0.0.1:52595/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 428854 T1395 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 428858 T1395 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 428873 T1483 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=4&maxShardsPerNode=2&name=testOverseerCol&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 428877 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428886 T1419 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 428888 T1419 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 428890 T1419 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 428890 T1419 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 428890 T1419 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 428906 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428913 T1418 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 428915 T1418 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 428915 T1418 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 428928 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 428936 T1505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 428936 T1449 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 428936 T1477 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 428938 T1417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 428939 T1422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 428940 T1435 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 428941 T1463 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 428940 T1491 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 428943 T1508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 428945 T1510 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429008 T1419 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 429008 T1419 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:52611_
   [junit4]   2> 429015 T1419 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:52614_
   [junit4]   2> 429017 T1419 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:52598_
   [junit4]   2> 429025 T1419 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:52617_
   [junit4]   2> 429027 T1419 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:52602_
   [junit4]   2> 429029 T1419 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:52605_
   [junit4]   2> 429031 T1419 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:52608_
   [junit4]   2> 429039 T1419 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:52611_
   [junit4]   2> 429042 T1409 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 429045 T1485 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 429046 T1469 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 429050 T1441 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 429051 T1457 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 429059 T1426 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 429060 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429060 T1409 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 429060 T1441 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 429059 T1485 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 429064 T1468 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 429068 T1457 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 429062 T1500 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 429060 T1469 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 429070 T1426 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 429069 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429074 T1468 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 429075 T1500 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 429077 T1418 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 429080 T1418 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52611",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52611_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 429111 T1418 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52598",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52598_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 429128 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429133 T1418 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52614",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52614_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 429153 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429158 T1418 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52605",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52605_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 429170 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429175 T1418 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52608",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52608_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 429188 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429192 T1418 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52602",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52602_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 429205 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429210 T1418 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52611",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52611_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 429225 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429239 T1418 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52617",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52617_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 429252 T1417 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 429362 T1449 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429363 T1491 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429363 T1510 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429364 T1505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429364 T1422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429364 T1508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429365 T1463 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429367 T1417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429365 T1435 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 429372 T1477 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 430062 T1409 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 430065 T1409 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1390052631922/testOverseerCol_shard2_replica1
   [junit4]   2> 430066 T1409 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 430066 T1441 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 430067 T1485 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 430069 T1441 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1390052634436/testOverseerCol_shard3_replica2
   [junit4]   2> 430069 T1441 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 430070 T1485 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1390052639449/testOverseerCol_shard1_replica2
   [junit4]   2> 430071 T1485 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 430072 T1469 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 430071 T1457 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 430078 T1457 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1390052636055/testOverseerCol_shard4_replica1
   [junit4]   2> 430079 T1457 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 430073 T1426 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 430080 T1469 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752/testOverseerCol_shard1_replica1
   [junit4]   2> 430081 T1469 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 430082 T1500 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 430083 T1468 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 430086 T1500 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1390052641057/testOverseerCol_shard2_replica2
   [junit4]   2> 430087 T1500 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 430088 T1409 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 430088 T1409 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 430082 T1426 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1390052632747/testOverseerCol_shard3_replica1
   [junit4]   2> 430090 T1426 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 430091 T1441 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 430092 T1441 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 430088 T1468 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752/testOverseerCol_shard4_replica2
   [junit4]   2> 430093 T1468 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 430093 T1485 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 430094 T1485 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 430095 T1469 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 430095 T1469 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 430096 T1457 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 430096 T1457 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 430101 T1500 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 430101 T1500 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 430102 T1426 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 430102 T1426 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 430103 T1468 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 430104 T1468 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 430114 T1485 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1390052639449/testOverseerCol_shard1_replica2/'
   [junit4]   2> 430118 T1457 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1390052636055/testOverseerCol_shard4_replica1/'
   [junit4]   2> 430125 T1469 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752/testOverseerCol_shard1_replica1/'
   [junit4]   2> 430132 T1441 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1390052634436/testOverseerCol_shard3_replica2/'
   [junit4]   2> 430149 T1426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1390052632747/testOverseerCol_shard3_replica1/'
   [junit4]   2> 430153 T1409 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1390052631922/testOverseerCol_shard2_replica1/'
   [junit4]   2> 430153 T1468 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1390052637752/testOverseerCol_shard4_replica2/'
   [junit4]   2> 430158 T1500 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1390052641057/testOverseerCol_shard2_replica2/'
   [junit4]   2> 430416 T1469 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 430447 T1485 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 430472 T1441 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 430507 T1426 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 430521 T1457 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 430523 T1468 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 430571 T1409 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 430644 T1500 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 430670 T1469 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 430686 T1469 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 430716 T1469 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 430781 T1457 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 430792 T1485 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 430794 T1468 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 430805 T1457 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 430811 T1468 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 430819 T1441 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 430821 T1485 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 430826 T1426 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 430826 T1441 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 430843 T1426 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 430862 T1485 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 430872 T1426 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 430880 T1457 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 430883 T1468 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 430888 T1441 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 430965 T1409 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 430973 T1409 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 430994 T1409 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 430997 T1500 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 431006 T1500 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 431026 T1500 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 433402 T1469 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433406 T1469 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433407 T1469 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 433432 T1469 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 433450 T1469 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 433521 T1500 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433522 T1500 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433523 T1500 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 433543 T1500 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 433549 T1500 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 433664 T1468 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433665 T1468 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433666 T1468 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 433665 T1485 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433669 T1485 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433671 T1485 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 433705 T1468 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 433719 T1468 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 433746 T1469 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 433752 T1485 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 433769 T1485 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 433782 T1441 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433785 T1441 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 433786 T1441 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 433784 T1469 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 433805 T1441 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 433809 T1441 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 433830 T1469 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 433831 T1500 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 433852 T1469 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 433852 T1469 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 433852 T1469 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 433855 T1500 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 433861 T1469 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 433861 T1469 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 433862 T1469 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory


[...truncated too long message...]

.close [awholynewcollection_0_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5332ca41
   [junit4]   2> 2149856 T6743 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=307,transaction_logs_total_number=1}
   [junit4]   2> 2149856 T6743 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 2149856 T6743 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2149857 T6743 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2149858 T6743 oasc.SolrCore.closeSearcher [awholynewcollection_0_shard1_replica1] Closing main searcher on request.
   [junit4]   2> 2149858 T6743 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2149858 T6743 oasc.CachingDirectoryFactory.closeCacheValue looking to close ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390053940381/awholynewcollection_0_shard1_replica1/data/index [CachedDir<<refCount=0;path=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390053940381/awholynewcollection_0_shard1_replica1/data/index;done=false>>]
   [junit4]   2> 2149859 T6743 oasc.CachingDirectoryFactory.close Closing directory: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390053940381/awholynewcollection_0_shard1_replica1/data/index
   [junit4]   2> 2149859 T6743 oasc.CachingDirectoryFactory.closeCacheValue looking to close ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390053940381/awholynewcollection_0_shard1_replica1/data [CachedDir<<refCount=0;path=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390053940381/awholynewcollection_0_shard1_replica1/data;done=false>>]
   [junit4]   2> 2149860 T6743 oasc.CachingDirectoryFactory.close Closing directory: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1390053940381/awholynewcollection_0_shard1_replica1/data
   [junit4]   2> 2149860 T6743 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2149860 T6743 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2149861 T6743 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2149861 T6743 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2149861 T6743 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> NOTE: test params are: codec=Lucene46: {timestamp=MockFixedIntBlock(blockSize=38), range_facet_l=MockFixedIntBlock(blockSize=38), id=PostingsFormat(name=Direct), range_facet_sl=PostingsFormat(name=MockSep), a_t=MockFixedIntBlock(blockSize=38), text=PostingsFormat(name=MockSep), range_facet_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), _version_=MockFixedIntBlock(blockSize=38), other_tl1=MockFixedIntBlock(blockSize=38), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), intDefault=MockFixedIntBlock(blockSize=38)}, docValues:{}, sim=DefaultSimilarity, locale=fr, timezone=Greenwich
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Apple Inc. 1.6.0_65 (64-bit)/cpus=2,threads=1,free=111350120,total=339935232
   [junit4]   2> NOTE: All tests run in this JVM: [AssignTest, AliasIntegrationTest, TestFieldCollectionResource, BasicZkTest, UpdateParamsTest, UpdateRequestProcessorFactoryTest, TestLMDirichletSimilarityFactory, CursorPagingTest, TimeZoneUtilsTest, DeleteReplicaTest, PingRequestHandlerTest, OverseerCollectionProcessorTest, ConnectionManagerTest, TestSolrQueryParserResource, ShardRoutingCustomTest, DocValuesMultiTest, TestSolrQueryParserDefaultOperatorResource, UniqFieldsUpdateProcessorFactoryTest, DebugComponentTest, DistanceFunctionTest, TestImplicitCoreProperties, AutoCommitTest, JsonLoaderTest, NoCacheHeaderTest, CopyFieldTest, TestCloudManagedSchema, TestSystemIdResolver, AddSchemaFieldsUpdateProcessorFactoryTest, TestValueSourceCache, AlternateDirectoryTest, TestFieldResource, TestCursorMarkWithoutUniqueKey, ChaosMonkeySafeLeaderTest, SolrTestCaseJ4Test, TestQuerySenderListener, TestRecovery, TestManagedSchema, TestAtomicUpdateErrorCases, TestRandomDVFaceting, TestCodecSupport, OverseerRolesTest, SpellPossibilityIteratorTest, BasicDistributedZk2Test, TestSort, TestClassNameShortening, TestRecoveryHdfs, TestPseudoReturnFields, TestPostingsSolrHighlighter, ReturnFieldsTest, SuggesterFSTTest, TestCharFilters, TestFiltering, TestQueryUtils, HdfsLockFactoryTest, BlockCacheTest, TestSchemaSimilarityResource, TestAnalyzeInfixSuggestions, TestReplicationHandler, TestWriterPerf, PeerSyncTest, PreAnalyzedUpdateProcessorTest, TestFuzzyAnalyzedSuggestions, TriLevelCompositeIdRoutingTest, InfoHandlerTest, ZkControllerTest, TestMultiCoreConfBootstrap, CacheHeaderTest, TestNumberUtils, TestFieldTypeCollectionResource, TestMergePolicyConfig, TestStressReorder, TestBM25SimilarityFactory, SimplePostToolTest, SuggesterWFSTTest, SolrRequestParserTest, ConvertedLegacyTest, ScriptEngineTest, TestCSVLoader, MoreLikeThisHandlerTest, TestGroupingSearch, TestSerializedLuceneMatchVersion, IndexSchemaRuntimeFieldTest, SpellCheckCollatorTest, TestCustomSort, HdfsSyncSliceTest, FieldAnalysisRequestHandlerTest, TestRTGBase, SolrIndexConfigTest, TestSurroundQueryParser, SolrIndexSplitterTest, UnloadDistributedZkTest, TestSolrXmlPersistor, HdfsCollectionsAPIDistributedZkTest, TestStressUserVersions, TestDocBasedVersionConstraints, ResponseLogComponentTest, TestSuggestSpellingConverter, DeleteShardTest, TestWordDelimiterFilterFactory, SuggesterTest, ZkNodePropsTest, DirectUpdateHandlerOptimizeTest, ChaosMonkeyNothingIsSafeTest, RAMDirectoryFactoryTest, TestReloadAndDeleteDocs, ShardRoutingTest, TestFoldingMultitermQuery, TestFileDictionaryLookup, TestStressRecovery, TestNoOpRegenerator, TestJmxMonitoredMap, QueryElevationComponentTest, TestComponentsName, TestRandomFaceting, TestConfig, LeaderElectionTest, TestFastLRUCache, ZkSolrClientTest, DistributedDebugComponentTest, BasicDistributedZkTest, TestDefaultSearchFieldResource, TestArbitraryIndexDir, TestRandomMergePolicy, TestSolrIndexConfig, DOMUtilTest, SolrCoreCheckLockOnStartupTest, LeaderElectionIntegrationTest, TestCoreContainer, TestSchemaVersionResource, DateMathParserTest, StatelessScriptUpdateProcessorFactoryTest, TestDynamicFieldResource, NotRequiredUniqueKeyTest, SpellingQueryConverterTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestSchemaNameResource, TestDistributedSearch, ClusterStateTest, TestAnalyzedSuggestions, RecoveryZkTest, LegacyHTMLStripCharFilterTest, TestFunctionQuery, TestCloudManagedSchemaAddField, TestQueryTypes, AnalysisAfterCoreReloadTest, BadIndexSchemaTest, TestCollationFieldDocValues, OutputWriterTest, TestManagedSchemaFieldResource, HdfsBasicDistributedZk2Test, OverseerTest, TestLFUCache, XsltUpdateRequestHandlerTest, TestDFRSimilarityFactory, DistributedSuggestComponentTest, PolyFieldTest, TestCSVResponseWriter, XmlUpdateRequestHandlerTest, FileBasedSpellCheckerTest, TestCollapseQParserPlugin, HdfsBasicDistributedZkTest, TestStressLucene, SpatialFilterTest, TestDefaultSimilarityFactory, MinimalSchemaTest, PrimitiveFieldTypeTest, BasicFunctionalityTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed in 430.16s, 1 test, 1 failure <<< FAILURES!

[...truncated 552 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1541 tests, 2 failures, 39 ignored (5 assumptions)

Total time: 87 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1208 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1208/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=5059, name=Timer-1, state=WAITING, group=TGRP-BasicDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=5059, name=Timer-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([8DC34AD19DFE2B19]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=5059, name=Timer-1, state=WAITING, group=TGRP-BasicDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=5059, name=Timer-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([8DC34AD19DFE2B19]:0)




Build Log:
[...truncated 10452 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> 1405559 T4710 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1405567 T4710 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1390033359466
   [junit4]   2> 1405569 T4710 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1405570 T4711 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1405670 T4710 oasc.ZkTestServer.run start zk server on port:53822
   [junit4]   2> 1405673 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1405684 T4717 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1255c465 name:ZooKeeperConnection Watcher:127.0.0.1:53822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1405685 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1405685 T4710 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1405698 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1405703 T4719 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7df80e29 name:ZooKeeperConnection Watcher:127.0.0.1:53822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1405704 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1405704 T4710 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1405713 T4710 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1405720 T4710 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1405730 T4710 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1405738 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1405739 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1405752 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1405753 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1405763 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1405763 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1405773 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1405773 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1405783 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1405784 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1405806 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1405807 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1405815 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1405816 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1405825 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1405826 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1405834 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1405835 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1405847 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1405849 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1405858 T4710 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1405858 T4710 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1406251 T4710 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1406257 T4710 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53825
   [junit4]   2> 1406259 T4710 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1406259 T4710 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1406260 T4710 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1390033359768
   [junit4]   2> 1406260 T4710 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1390033359768/'
   [junit4]   2> 1406301 T4710 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1390033359768/solr.xml
   [junit4]   2> 1406398 T4710 oasc.CoreContainer.<init> New CoreContainer 1961715192
   [junit4]   2> 1406398 T4710 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1390033359768/]
   [junit4]   2> 1406400 T4710 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1406401 T4710 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1406401 T4710 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1406401 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1406401 T4710 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1406402 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1406402 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1406402 T4710 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1406403 T4710 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1406413 T4710 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1406413 T4710 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1406413 T4710 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1406414 T4710 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53822/solr
   [junit4]   2> 1406414 T4710 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1406416 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1406420 T4730 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dc1b8c1 name:ZooKeeperConnection Watcher:127.0.0.1:53822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1406421 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1406426 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1406431 T4732 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19afde47 name:ZooKeeperConnection Watcher:127.0.0.1:53822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1406432 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1406440 T4710 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1406450 T4710 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1406459 T4710 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1406464 T4710 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53825_
   [junit4]   2> 1406467 T4710 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53825_
   [junit4]   2> 1406478 T4710 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1406484 T4710 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1406497 T4710 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1406504 T4710 oasc.Overseer.start Overseer (id=91097226246488067-127.0.0.1:53825_-n_0000000000) starting
   [junit4]   2> 1406513 T4710 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1406524 T4734 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1406526 T4734 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1406526 T4710 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1406534 T4710 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1406541 T4710 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1406549 T4733 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1406563 T4735 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1406563 T4735 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1406566 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1406566 T4735 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1406570 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1406572 T4733 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53825",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53825_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1406572 T4733 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1406572 T4733 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1406579 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1406581 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1407568 T4735 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1407568 T4735 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1390033359768/collection1
   [junit4]   2> 1407568 T4735 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1407571 T4735 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1407571 T4735 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1407574 T4735 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1390033359768/collection1/'
   [junit4]   2> 1407576 T4735 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1390033359768/collection1/lib/classes/' to classloader
   [junit4]   2> 1407577 T4735 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1390033359768/collection1/lib/README' to classloader
   [junit4]   2> 1407680 T4735 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1407859 T4735 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1407863 T4735 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1407880 T4735 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1408865 T4735 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1408866 T4735 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1408867 T4735 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1408877 T4735 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1408881 T4735 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1408921 T4735 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1408930 T4735 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1408940 T4735 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1408943 T4735 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1408943 T4735 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1408944 T4735 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1408947 T4735 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1408947 T4735 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1408947 T4735 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1408948 T4735 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1390033359768/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/control/data/
   [junit4]   2> 1408948 T4735 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c9df04e
   [junit4]   2> 1408949 T4735 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/control/data
   [junit4]   2> 1408950 T4735 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/control/data/index/
   [junit4]   2> 1408950 T4735 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1408951 T4735 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/control/data/index
   [junit4]   2> 1408951 T4735 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4688339554564317]
   [junit4]   2> 1408953 T4735 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@38c05c31 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4ae7c654),segFN=segments_1,generation=1}
   [junit4]   2> 1408954 T4735 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1408960 T4735 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1408961 T4735 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1408962 T4735 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1408962 T4735 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1408963 T4735 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1408963 T4735 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1408964 T4735 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1408965 T4735 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1408965 T4735 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1408966 T4735 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1408967 T4735 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1408967 T4735 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1408968 T4735 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1408969 T4735 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1408969 T4735 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1408970 T4735 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1408984 T4735 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1408992 T4735 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1408992 T4735 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1408993 T4735 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=22.419921875, floorSegmentMB=1.6259765625, forceMergeDeletesPctAllowed=20.9466656940915, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1408994 T4735 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@38c05c31 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4ae7c654),segFN=segments_1,generation=1}
   [junit4]   2> 1408994 T4735 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1408995 T4735 oass.SolrIndexSearcher.<init> Opening Searcher@6ea408a9 main
   [junit4]   2> 1409000 T4736 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ea408a9 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1409001 T4735 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1409003 T4710 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1409004 T4710 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1409003 T4739 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53825 collection:control_collection shard:shard1
   [junit4]   2> 1409007 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1409009 T4739 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1409012 T4741 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@587dcfc0 name:ZooKeeperConnection Watcher:127.0.0.1:53822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1409013 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1409018 T4710 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1409025 T4710 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1409029 T4739 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1409039 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1409044 T4739 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1409045 T4739 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C981 name=collection1 org.apache.solr.core.SolrCore@103079d url=http://127.0.0.1:53825/collection1 node=127.0.0.1:53825_ C981_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:53825, core=collection1, node_name=127.0.0.1:53825_}
   [junit4]   2> 1409045 T4739 C981 P53825 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53825/collection1/
   [junit4]   2> 1409045 T4739 C981 P53825 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1409046 T4739 C981 P53825 oasc.SyncStrategy.syncToMe http://127.0.0.1:53825/collection1/ has no replicas
   [junit4]   2> 1409046 T4739 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53825/collection1/ shard1
   [junit4]   2> 1409047 T4739 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1409053 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1409070 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1409074 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1409075 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1409099 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1409107 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1409217 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1409217 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1409236 T4739 oasc.ZkController.register We are http://127.0.0.1:53825/collection1/ and leader is http://127.0.0.1:53825/collection1/
   [junit4]   2> 1409237 T4739 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53825
   [junit4]   2> 1409237 T4739 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1409238 T4739 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1409238 T4739 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1409240 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1409241 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1409241 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1409242 T4739 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1409246 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1409248 T4733 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53825",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53825_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1409255 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1409364 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1409364 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1409506 T4710 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1409512 T4710 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53829
   [junit4]   2> 1409515 T4710 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1409516 T4710 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1409516 T4710 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1390033362924
   [junit4]   2> 1409517 T4710 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1390033362924/'
   [junit4]   2> 1409578 T4710 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1390033362924/solr.xml
   [junit4]   2> 1409695 T4710 oasc.CoreContainer.<init> New CoreContainer 2135080273
   [junit4]   2> 1409695 T4710 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1390033362924/]
   [junit4]   2> 1409698 T4710 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1409699 T4710 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1409699 T4710 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1409700 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1409700 T4710 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1409701 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1409701 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1409701 T4710 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1409702 T4710 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1409717 T4710 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1409718 T4710 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1409718 T4710 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1409719 T4710 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53822/solr
   [junit4]   2> 1409719 T4710 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1409721 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1409730 T4752 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@769b521c name:ZooKeeperConnection Watcher:127.0.0.1:53822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1409730 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1409746 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1409753 T4754 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f8f0a16 name:ZooKeeperConnection Watcher:127.0.0.1:53822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1409753 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1409774 T4710 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1410789 T4710 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53829_
   [junit4]   2> 1410792 T4710 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53829_
   [junit4]   2> 1410800 T4741 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1410800 T4754 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1410800 T4732 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1410823 T4755 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1410823 T4755 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1410826 T4755 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1410826 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1410827 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1410828 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1410832 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1410834 T4733 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53829",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53829_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1410835 T4733 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1410835 T4733 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1410844 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1410952 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1410952 T4754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1410952 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1411828 T4755 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1411828 T4755 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1390033362924/collection1
   [junit4]   2> 1411829 T4755 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1411832 T4755 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1411832 T4755 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1411835 T4755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1390033362924/collection1/'
   [junit4]   2> 1411837 T4755 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1390033362924/collection1/lib/classes/' to classloader
   [junit4]   2> 1411838 T4755 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1390033362924/collection1/lib/README' to classloader
   [junit4]   2> 1411944 T4755 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1412040 T4755 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1412044 T4755 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1412060 T4755 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1412930 T4755 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1412931 T4755 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1412931 T4755 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1412945 T4755 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1412952 T4755 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1412994 T4755 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1413003 T4755 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1413012 T4755 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1413016 T4755 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1413016 T4755 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1413016 T4755 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1413019 T4755 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1413020 T4755 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1413021 T4755 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1413021 T4755 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1390033362924/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty1/
   [junit4]   2> 1413021 T4755 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c9df04e
   [junit4]   2> 1413022 T4755 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty1
   [junit4]   2> 1413023 T4755 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty1/index/
   [junit4]   2> 1413023 T4755 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1413024 T4755 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty1/index
   [junit4]   2> 1413025 T4755 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4688339554564317]
   [junit4]   2> 1413026 T4755 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@78729fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7f47799b),segFN=segments_1,generation=1}
   [junit4]   2> 1413027 T4755 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1413033 T4755 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1413035 T4755 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1413035 T4755 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1413036 T4755 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1413036 T4755 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1413037 T4755 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1413037 T4755 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1413038 T4755 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1413038 T4755 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1413039 T4755 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1413040 T4755 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1413041 T4755 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1413041 T4755 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1413042 T4755 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1413043 T4755 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1413044 T4755 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1413060 T4755 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1413068 T4755 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1413069 T4755 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1413070 T4755 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=22.419921875, floorSegmentMB=1.6259765625, forceMergeDeletesPctAllowed=20.9466656940915, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1413071 T4755 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@78729fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7f47799b),segFN=segments_1,generation=1}
   [junit4]   2> 1413071 T4755 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1413071 T4755 oass.SolrIndexSearcher.<init> Opening Searcher@665f48af main
   [junit4]   2> 1413083 T4756 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@665f48af main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1413088 T4755 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1413090 T4710 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1413090 T4710 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1413090 T4759 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53829 collection:collection1 shard:shard2
   [junit4]   2> 1413094 T4759 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1413114 T4759 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1413122 T4759 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1413122 T4759 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C982 name=collection1 org.apache.solr.core.SolrCore@4cec1610 url=http://127.0.0.1:53829/collection1 node=127.0.0.1:53829_ C982_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:53829, core=collection1, node_name=127.0.0.1:53829_}
   [junit4]   2> 1413122 T4759 C982 P53829 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53829/collection1/
   [junit4]   2> 1413123 T4759 C982 P53829 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1413123 T4759 C982 P53829 oasc.SyncStrategy.syncToMe http://127.0.0.1:53829/collection1/ has no replicas
   [junit4]   2> 1413123 T4759 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53829/collection1/ shard2
   [junit4]   2> 1413124 T4759 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1413129 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1413129 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1413130 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1413136 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1413155 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1413157 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1413161 T4754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1413164 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1413180 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1413189 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1413294 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1413295 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1413298 T4754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1413325 T4759 oasc.ZkController.register We are http://127.0.0.1:53829/collection1/ and leader is http://127.0.0.1:53829/collection1/
   [junit4]   2> 1413326 T4759 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53829
   [junit4]   2> 1413326 T4759 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1413327 T4759 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1413327 T4759 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1413332 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1413332 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1413332 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1413333 T4759 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1413338 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1413340 T4733 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53829",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53829_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1413349 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1413502 T4754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1413502 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1413503 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1413629 T4710 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1413635 T4710 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53832
   [junit4]   2> 1413638 T4710 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1413638 T4710 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1413639 T4710 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1390033366989
   [junit4]   2> 1413639 T4710 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1390033366989/'
   [junit4]   2> 1413683 T4710 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1390033366989/solr.xml
   [junit4]   2> 1413768 T4710 oasc.CoreContainer.<init> New CoreContainer 187307176
   [junit4]   2> 1413769 T4710 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1390033366989/]
   [junit4]   2> 1413770 T4710 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1413771 T4710 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1413771 T4710 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1413771 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1413772 T4710 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1413772 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1413772 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1413773 T4710 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1413773 T4710 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1413783 T4710 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1413784 T4710 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1413785 T4710 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1413785 T4710 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53822/solr
   [junit4]   2> 1413786 T4710 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1413787 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1413791 T4770 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77b22751 name:ZooKeeperConnection Watcher:127.0.0.1:53822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1413792 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1413798 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1413802 T4772 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38dbbbdb name:ZooKeeperConnection Watcher:127.0.0.1:53822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1413803 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1413813 T4710 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1414825 T4710 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53832_
   [junit4]   2> 1414829 T4710 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53832_
   [junit4]   2> 1414837 T4772 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1414838 T4741 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1414838 T4754 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1414838 T4732 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1414865 T4773 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1414866 T4773 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1414869 T4773 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1414869 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1414870 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1414870 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1414876 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1414878 T4733 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53832",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53832_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1414879 T4733 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1414879 T4733 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1414887 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1415000 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1415000 T4754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1415000 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1415000 T4772 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1415871 T4773 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1415871 T4773 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1390033366989/collection1
   [junit4]   2> 1415872 T4773 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1415874 T4773 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1415874 T4773 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1415877 T4773 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1390033366989/collection1/'
   [junit4]   2> 1415879 T4773 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1390033366989/collection1/lib/classes/' to classloader
   [junit4]   2> 1415879 T4773 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1390033366989/collection1/lib/README' to classloader
   [junit4]   2> 1415964 T4773 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1416083 T4773 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1416087 T4773 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1416102 T4773 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1416797 T4773 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1416797 T4773 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1416798 T4773 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1416808 T4773 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1416812 T4773 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1416844 T4773 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1416853 T4773 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1416861 T4773 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1416865 T4773 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1416865 T4773 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1416865 T4773 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1416869 T4773 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1416869 T4773 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1416869 T4773 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1416870 T4773 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1390033366989/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty2/
   [junit4]   2> 1416870 T4773 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c9df04e
   [junit4]   2> 1416871 T4773 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty2
   [junit4]   2> 1416872 T4773 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty2/index/
   [junit4]   2> 1416872 T4773 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1416872 T4773 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1390033359466/jetty2/index
   [junit4]   2> 1416874 T4773 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4688339554564317]
   [junit4]   2> 1416876 T4773 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6c70918 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7e19d5df),segFN=segments_1,generation=1}
   [junit4]   2> 1416877 T4773 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1416882 T4773 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1416883 T4773 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1416883 T4773 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1416884 T4773 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1416884 T4773 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1416884 T4773 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1416885 T4773 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1416885 T4773 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1416886 T4773 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1416886 T4773 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1416888 T4773 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1416888 T4773 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1416889 T4773 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1416890 T4773 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1416890 T4773 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1416891 T4773 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1416901 T4773 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1416906 T4773 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1416907 T4773 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1416907 T4773 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=22.419921875, floorSegmentMB=1.6259765625, forceMergeDeletesPctAllowed=20.9466656940915, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1416908 T4773 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6c70918 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7e19d5df),segFN=segments_1,generation=1}
   [junit4]   2> 1416908 T4773 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1416909 T4773 oass.SolrIndexSearcher.<init> Opening Searcher@4c2635ce main
   [junit4]   2> 1416913 T4774 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c2635ce main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1416915 T4773 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1416918 T4710 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1416918 T4710 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1416918 T4777 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53832 collection:collection1 shard:shard1
   [junit4]   2> 1416921 T4777 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1416938 T4777 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1416943 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1416943 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1416943 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1416945 T4777 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1416945 T4777 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C983 name=collection1 org.apache.solr.core.SolrCore@64ee4e03 url=http://127.0.0.1:53832/collection1 node=127.0.0.1:53832_ C983_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:53832, core=collection1, node_name=127.0.0.1:53832_}
   [junit4]   2> 1416946 T4777 C983 P53832 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53832/collection1/
   [junit4]   2> 1416946 T4777 C983 P53832 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1416946 T4777 C983 P53832 oasc.SyncStrategy.syncToMe http://127.0.0.1:53832/collection1/ has no replicas
   [junit4]   2> 1416946 T4777 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53832/collection1/ shard1
   [junit4]   2> 1416947 T4777 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1416949 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1416962 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1416964 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1416965 T4772 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1416966 T4754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1416966 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1416977 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1416984 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417095 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1417095 T4754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1417096 T4772 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1417098 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1417117 T4777 oasc.ZkController.register We are http://127.0.0.1:53832/collection1/ and leader is http://127.0.0.1:53832/collection1/
   [junit4]   2> 1417117 T4777 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53832
   [junit4]   2> 1417117 T4777 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1417118 T4777 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1417118 T4777 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1417121 T4777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1417122 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417122 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417122 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417125 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1417126 T4733 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53832",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53832_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1417132 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417237 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1417237 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1417238 T4754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1417238 T4772 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1417256 T4710 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1417261 T4710 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53835
   [junit4]   2> 1417263 T4710 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1417264 T4710 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1417264 T4710 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1390033370817
   [junit4]   2> 1417264 T4710 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1390033370817/'
   [junit4]   2> 1417301 T4710 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1390033370817/solr.xml
   [junit4]   2> 1417377 T4710 oasc.CoreContainer.<init> New CoreContainer 2062070657
   [junit4]   2> 1417377 T4710 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1390033370817/]
   [junit4]   2> 1417379 T4710 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1417379 T4710 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1417379 T4710 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1417380 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1417380 T4710 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1417380 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1417381 T4710 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1417381 T4710 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1417382 T4710 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1417391 T4710 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1417391 T4710 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1417392 T4710 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1417392 T4710 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53822/solr
   [junit4]   2> 1417392 T4710 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1417394 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1417398 T4788 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43611931 name:ZooKeeperConnection Watcher:127.0.0.1:53822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1417398 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1417403 T4710 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1417408 T4790 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70cc8584 name:ZooKeeperConnection Watcher:127.0.0.1:53822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1417408 T4710 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1417417 T4710 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1418426 T4710 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53835_
   [junit4]   2> 1418429 T4710 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53835_
   [junit4]   2> 1418438 T4790 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1418438 T4754 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1418439 T4741 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1418439 T4732 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1418438 T4772 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1418461 T4791 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1418462 T4791 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1418464 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1418465 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1418466 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1418464 T4791 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1418470 T4733 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1418472 T4733 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53835",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53835_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1418473 T4733 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1418473 T4733 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1418481 T4732 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1418588 T4754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1418588 T4741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1418588 T4790 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1418589 T4772 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1418589 T4732 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1419468 T4791 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1419469 T4791 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1390033370817/collection1
   [junit4]   2> 1419469 T4791 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1419471 T4791 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1419471 T4791 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1419475 T4791 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1390033370817/collection1/'
   [junit4]   2> 1419477 T4791 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1390033370817/collection1/lib/classes/' to classloader
   [junit4]   2> 1419477 T4791 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1390033370817/collection1/lib/README' to classloader
   [junit4]   2> 1419565 T4791 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1419652 T4791 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1419656 T4791 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1419666 T4791 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1420395 T4791 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1420395 T4791 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1420396 T4791 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1420406 T4791 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1420410 T4791 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1420441 T4791 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1420447 T4791 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1420453 T4791 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1420456 T4791 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1420456 T4791 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1420457 T4791 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1420459 T4791 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1420460 T4791 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1420460 T4791 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1420460 T4791 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1390033370817/collection1/, dataDir=

[...truncated too long message...]

Cnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1522873 T4809 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1522873 T4809 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1522873 T4809 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1522873 T4809 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 117327 T4709 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 137354 T4709 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]   2> 	   1) Thread[id=5059, name=Timer-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 137355 T4709 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=5059, name=Timer-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 140377 T4709 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=5059, name=Timer-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=en_US, timezone=Navajo
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=2,free=143458968,total=448790528
   [junit4]   2> NOTE: All tests run in this JVM: [TestBinaryResponseWriter, SystemInfoHandlerTest, TriLevelCompositeIdRoutingTest, XmlUpdateRequestHandlerTest, TestSolrXml, TermVectorComponentDistributedTest, CSVRequestHandlerTest, InfoHandlerTest, RequiredFieldsTest, TestCSVLoader, TestFunctionQuery, StressHdfsTest, TestHashPartitioner, LeaderElectionTest, TestFaceting, TestCSVResponseWriter, AddSchemaFieldsUpdateProcessorFactoryTest, HdfsChaosMonkeySafeLeaderTest, TestSerializedLuceneMatchVersion, TestSort, TermsComponentTest, TestQueryTypes, TestSolrDeletionPolicy1, RequestHandlersTest, TestRandomFaceting, SearchHandlerTest, TestSchemaResource, BlockCacheTest, TestRealTimeGet, TestDFRSimilarityFactory, TestRandomDVFaceting, NotRequiredUniqueKeyTest, SampleTest, JsonLoaderTest, TestValueSourceCache, SolrTestCaseJ4Test, BasicFunctionalityTest, TestMultiCoreConfBootstrap, TestManagedSchema, TestFileDictionaryLookup, SignatureUpdateProcessorFactoryTest, SpatialFilterTest, TestMergePolicyConfig, HardAutoCommitTest, TestCoreDiscovery, TestRangeQuery, TestSearchPerf, TestFieldTypeCollectionResource, DistributedDebugComponentTest, TestManagedSchemaFieldResource, DirectSolrConnectionTest, TestCharFilters, DocValuesMultiTest, TestSweetSpotSimilarityFactory, TestStressUserVersions, BadIndexSchemaTest, ShardRoutingTest, FileBasedSpellCheckerTest, TestCustomSort, OverseerTest, IndexSchemaRuntimeFieldTest, ConvertedLegacyTest, ShowFileRequestHandlerTest, TestLRUCache, ConnectionManagerTest, TestNRTOpen, HighlighterTest, CopyFieldTest, TestPseudoReturnFields, TestDocBasedVersionConstraints, SpellCheckCollatorTest, TestLazyCores, DistanceFunctionTest, DocValuesMissingTest, ClusterStateUpdateTest, OpenExchangeRatesOrgProviderTest, PeerSyncTest, DirectUpdateHandlerOptimizeTest, DistributedQueryComponentCustomSortTest, SolrIndexConfigTest, DistributedTermsComponentTest, SolrRequestParserTest, HdfsSyncSliceTest, TestQuerySenderListener, BlockDirectoryTest, DistributedQueryElevationComponentTest, HdfsCollectionsAPIDistributedZkTest, CoreAdminCreateDiscoverTest, DocumentBuilderTest, TestCoreContainer, QueryEqualityTest, DOMUtilTest, DefaultValueUpdateProcessorTest, PluginInfoTest, BadCopyFieldTest, ExternalFileFieldSortTest, TestFieldTypeResource, DateFieldTest, ZkCLITest, TestRemoteStreaming, MinimalSchemaTest, TestCollationFieldDocValues, SolrCoreCheckLockOnStartupTest, TestFastOutputStream, DistribCursorPagingTest, DebugComponentTest, RecoveryZkTest, FieldMutatingUpdateProcessorTest, RAMDirectoryFactoryTest, TestSurroundQueryParser, TestReload, SpellPossibilityIteratorTest, TestLFUCache, TestAnalyzedSuggestions, TestJmxIntegration, SimpleFacetsTest, FieldAnalysisRequestHandlerTest, WordBreakSolrSpellCheckerTest, TestPostingsSolrHighlighter, TimeZoneUtilsTest, CurrencyFieldXmlFileTest, TestSolrDeletionPolicy2, HdfsBasicDistributedZk2Test, TestSuggestSpellingConverter, TestQueryUtils, TestInfoStreamLogging, ScriptEngineTest, MBeansHandlerTest, MigrateRouteKeyTest, PrimUtilsTest, TestSolrJ, TestElisionMultitermQuery, TestSchemaVersionResource, TestCodecSupport, TestAnalyzeInfixSuggestions, CollectionsAPIDistributedZkTest, TestReversedWildcardFilterFactory, TestCursorMarkWithoutUniqueKey, FileUtilsTest, StatsComponentTest, TestAtomicUpdateErrorCases, TestUniqueKeyFieldResource, TestDistributedSearch, UnloadDistributedZkTest, FullSolrCloudDistribCmdsTest, QueryResultKeyTest, BadComponentTest, TestCopyFieldCollectionResource, TestSimpleQParserPlugin, BasicDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.seed=8DC34AD19DFE2B19 -Dtests.slow=true -Dtests.locale=en_US -Dtests.timezone=Navajo -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s | BasicDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]    >    1) Thread[id=5059, name=Timer-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8DC34AD19DFE2B19]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=5059, name=Timer-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8DC34AD19DFE2B19]:0)
   [junit4] Completed in 140.48s, 1 test, 2 errors <<< FAILURES!

[...truncated 603 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1541 tests, 2 suite-level errors, 715 ignored

Total time: 61 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1207 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1207/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 16877 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140118_005511_620.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] 2014-01-18 00:55:49.823 java[513:6d03] Unable to load realm info from SCDynamicStore
   [junit4] <<< JVM J0: EOF ----

[...truncated 24830 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene...
  [javadoc] Loading source files for package org.apache.lucene.analysis...
  [javadoc] Loading source files for package org.apache.lucene.analysis.tokenattributes...
  [javadoc] Loading source files for package org.apache.lucene.codecs...
  [javadoc] Loading source files for package org.apache.lucene.codecs.compressing...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene3x...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene40...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene41...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene42...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene45...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene46...
  [javadoc] Loading source files for package org.apache.lucene.codecs.perfield...
  [javadoc] Loading source files for package org.apache.lucene.document...
  [javadoc] Loading source files for package org.apache.lucene.index...
  [javadoc] Loading source files for package org.apache.lucene.search...
  [javadoc] Loading source files for package org.apache.lucene.search.payloads...
  [javadoc] Loading source files for package org.apache.lucene.search.similarities...
  [javadoc] Loading source files for package org.apache.lucene.search.spans...
  [javadoc] Loading source files for package org.apache.lucene.store...
  [javadoc] Loading source files for package org.apache.lucene.util...
  [javadoc] Loading source files for package org.apache.lucene.util.automaton...
  [javadoc] Loading source files for package org.apache.lucene.util.fst...
  [javadoc] Loading source files for package org.apache.lucene.util.mutable...
  [javadoc] Loading source files for package org.apache.lucene.util.packed...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/core/org/apache/lucene/search/package-summary.html...
  [javadoc] Copying file /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/core/src/java/org/apache/lucene/search/doc-files/nrq-formula-1.png to directory /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/core/org/apache/lucene/search/doc-files...
  [javadoc] Copying file /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/core/src/java/org/apache/lucene/search/doc-files/nrq-formula-2.png to directory /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/core/org/apache/lucene/search/doc-files...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/core/help-doc.html...
  [javadoc] 1 warning

[...truncated 27 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.analysis.ar...
  [javadoc] Loading source files for package org.apache.lucene.analysis.bg...
  [javadoc] Loading source files for package org.apache.lucene.analysis.br...
  [javadoc] Loading source files for package org.apache.lucene.analysis.ca...
  [javadoc] Loading source files for package org.apache.lucene.analysis.charfilter...
  [javadoc] Loading source files for package org.apache.lucene.analysis.cjk...
  [javadoc] Loading source files for package org.apache.lucene.analysis.ckb...
  [javadoc] Loading source files for package org.apache.lucene.analysis.cn...
  [javadoc] Loading source files for package org.apache.lucene.analysis.commongrams...
  [javadoc] Loading source files for package org.apache.lucene.analysis.compound...
  [javadoc] Loading source files for package org.apache.lucene.analysis.compound.hyphenation...
  [javadoc] Loading source files for package org.apache.lucene.analysis.core...
  [javadoc] Loading source files for package org.apache.lucene.analysis.cz...
  [javadoc] Loading source files for package org.apache.lucene.analysis.da...
  [javadoc] Loading source files for package org.apache.lucene.analysis.de...
  [javadoc] Loading source files for package org.apache.lucene.analysis.el...
  [javadoc] Loading source files for package org.apache.lucene.analysis.en...
  [javadoc] Loading source files for package org.apache.lucene.analysis.es...
  [javadoc] Loading source files for package org.apache.lucene.analysis.eu...
  [javadoc] Loading source files for package org.apache.lucene.analysis.fa...
  [javadoc] Loading source files for package org.apache.lucene.analysis.fi...
  [javadoc] Loading source files for package org.apache.lucene.analysis.fr...
  [javadoc] Loading source files for package org.apache.lucene.analysis.ga...
  [javadoc] Loading source files for package org.apache.lucene.analysis.gl...
  [javadoc] Loading source files for package org.apache.lucene.analysis.hi...
  [javadoc] Loading source files for package org.apache.lucene.analysis.hu...
  [javadoc] Loading source files for package org.apache.lucene.analysis.hunspell...
  [javadoc] Loading source files for package org.apache.lucene.analysis.hy...
  [javadoc] Loading source files for package org.apache.lucene.analysis.id...
  [javadoc] Loading source files for package org.apache.lucene.analysis.in...
  [javadoc] Loading source files for package org.apache.lucene.analysis.it...
  [javadoc] Loading source files for package org.apache.lucene.analysis.lv...
  [javadoc] Loading source files for package org.apache.lucene.analysis.miscellaneous...
  [javadoc] Loading source files for package org.apache.lucene.analysis.ngram...
  [javadoc] Loading source files for package org.apache.lucene.analysis.nl...
  [javadoc] Loading source files for package org.apache.lucene.analysis.no...
  [javadoc] Loading source files for package org.apache.lucene.analysis.path...
  [javadoc] Loading source files for package org.apache.lucene.analysis.pattern...
  [javadoc] Loading source files for package org.apache.lucene.analysis.payloads...
  [javadoc] Loading source files for package org.apache.lucene.analysis.position...
  [javadoc] Loading source files for package org.apache.lucene.analysis.pt...
  [javadoc] Loading source files for package org.apache.lucene.analysis.query...
  [javadoc] Loading source files for package org.apache.lucene.analysis.reverse...
  [javadoc] Loading source files for package org.apache.lucene.analysis.ro...
  [javadoc] Loading source files for package org.apache.lucene.analysis.ru...
  [javadoc] Loading source files for package org.apache.lucene.analysis.shingle...
  [javadoc] Loading source files for package org.apache.lucene.analysis.sinks...
  [javadoc] Loading source files for package org.apache.lucene.analysis.snowball...
  [javadoc] Loading source files for package org.apache.lucene.analysis.standard...
  [javadoc] Loading source files for package org.apache.lucene.analysis.standard.std31...
  [javadoc] Loading source files for package org.apache.lucene.analysis.standard.std34...
  [javadoc] Loading source files for package org.apache.lucene.analysis.standard.std36...
  [javadoc] Loading source files for package org.apache.lucene.analysis.standard.std40...
  [javadoc] Loading source files for package org.apache.lucene.analysis.sv...
  [javadoc] Loading source files for package org.apache.lucene.analysis.synonym...
  [javadoc] Loading source files for package org.apache.lucene.analysis.th...
  [javadoc] Loading source files for package org.apache.lucene.analysis.tr...
  [javadoc] Loading source files for package org.apache.lucene.analysis.util...
  [javadoc] Loading source files for package org.apache.lucene.analysis.wikipedia...
  [javadoc] Loading source files for package org.apache.lucene.collation...
  [javadoc] Loading source files for package org.apache.lucene.collation.tokenattributes...
  [javadoc] Loading source files for package org.tartarus.snowball...
  [javadoc] Loading source files for package org.tartarus.snowball.ext...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/analyzers-common/help-doc.html...
  [javadoc] 1 warning

[...truncated 29 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.analysis.icu...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.analysis.icu.segmentation...
  [javadoc] Loading source files for package org.apache.lucene.analysis.icu.tokenattributes...
  [javadoc] Loading source files for package org.apache.lucene.collation...
  [javadoc] Loading source files for package org.apache.lucene.collation.tokenattributes...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/analyzers-icu/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 28 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.analysis.ja...
  [javadoc] Loading source files for package org.apache.lucene.analysis.ja.dict...
  [javadoc] Loading source files for package org.apache.lucene.analysis.ja.tokenattributes...
  [javadoc] Loading source files for package org.apache.lucene.analysis.ja.util...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/analyzers-kuromoji/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 28 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.analysis.morfologik...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/analyzers-morfologik/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 28 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.analysis.phonetic...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/analyzers-phonetic/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 28 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.analysis.cn.smart...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.analysis.cn.smart.hhmm...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/analyzers-smartcn/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 28 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.analysis.pl...
  [javadoc] Loading source files for package org.apache.lucene.analysis.stempel...
  [javadoc] Loading source files for package org.egothor.stemmer...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/analyzers-stempel/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 28 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.analysis.uima...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.analysis.uima.ae...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/analyzers-uima/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 31 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.index.memory...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/memory/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 30 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.search.highlight...
  [javadoc] Loading source files for package org.apache.lucene.search.postingshighlight...
  [javadoc] Loading source files for package org.apache.lucene.search.vectorhighlight...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/highlighter/help-doc.html...
  [javadoc] 1 warning

[...truncated 32 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.queries...
  [javadoc] Loading source files for package org.apache.lucene.queries.function...
  [javadoc] Loading source files for package org.apache.lucene.queries.function.docvalues...
  [javadoc] Loading source files for package org.apache.lucene.queries.function.valuesource...
  [javadoc] Loading source files for package org.apache.lucene.queries.mlt...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/queries/help-doc.html...
  [javadoc] 1 warning

[...truncated 27 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.sandbox.queries...
  [javadoc] Loading source files for package org.apache.lucene.sandbox.queries.regex...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/sandbox/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 27 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.queryparser.analyzing...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.queryparser.classic...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.complexPhrase...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.ext...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.builders...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.config...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.messages...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.nodes...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.parser...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.processors...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.util...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.messages...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.precedence...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.precedence.processors...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.builders...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.config...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.nodes...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.parser...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.processors...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.simple...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.surround.parser...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.surround.query...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.xml...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.xml.builders...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/queryparser/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 30 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.facet...
  [javadoc] Loading source files for package org.apache.lucene.facet.range...
  [javadoc] Loading source files for package org.apache.lucene.facet.sortedset...
  [javadoc] Loading source files for package org.apache.lucene.facet.taxonomy...
  [javadoc] Loading source files for package org.apache.lucene.facet.taxonomy.directory...
  [javadoc] Loading source files for package org.apache.lucene.facet.taxonomy.writercache...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/facet/help-doc.html...
  [javadoc] 1 warning

[...truncated 30 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.spatial...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.spatial.prefix...
  [javadoc] Loading source files for package org.apache.lucene.spatial.prefix.tree...
  [javadoc] Loading source files for package org.apache.lucene.spatial.query...
  [javadoc] Loading source files for package org.apache.lucene.spatial.util...
  [javadoc] Loading source files for package org.apache.lucene.spatial.vector...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/spatial/help-doc.html...
  [javadoc] 1 warning

[...truncated 35 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.benchmark...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.byTask...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.byTask.feeds...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.byTask.programmatic...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.byTask.stats...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.byTask.tasks...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.byTask.utils...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.quality...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.quality.trec...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.quality.utils...
  [javadoc] Loading source files for package org.apache.lucene.benchmark.utils...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/benchmark/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 30 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.classification...
  [javadoc] Loading source files for package org.apache.lucene.classification.utils...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/classification/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 25 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.codecs.appending...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.codecs.blockterms...
  [javadoc] Loading source files for package org.apache.lucene.codecs.bloom...
  [javadoc] Loading source files for package org.apache.lucene.codecs.diskdv...
  [javadoc] Loading source files for package org.apache.lucene.codecs.intblock...
  [javadoc] Loading source files for package org.apache.lucene.codecs.memory...
  [javadoc] Loading source files for package org.apache.lucene.codecs.pulsing...
  [javadoc] Loading source files for package org.apache.lucene.codecs.sep...
  [javadoc] Loading source files for package org.apache.lucene.codecs.simpletext...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/codecs/help-doc.html...
  [javadoc] 1 warning

[...truncated 37 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.expressions...
  [javadoc] Loading source files for package org.apache.lucene.expressions.js...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/expressions/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 36 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.demo...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.demo.facet...
  [javadoc] Loading source files for package org.apache.lucene.demo.xmlparser...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/demo/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 27 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.expressions...
  [javadoc] Loading source files for package org.apache.lucene.expressions.js...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/expressions/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 27 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.facet...
  [javadoc] Loading source files for package org.apache.lucene.facet.range...
  [javadoc] Loading source files for package org.apache.lucene.facet.sortedset...
  [javadoc] Loading source files for package org.apache.lucene.facet.taxonomy...
  [javadoc] Loading source files for package org.apache.lucene.facet.taxonomy.directory...
  [javadoc] Loading source files for package org.apache.lucene.facet.taxonomy.writercache...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/facet/help-doc.html...
  [javadoc] 1 warning

[...truncated 28 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.search.grouping...
  [javadoc] Loading source files for package org.apache.lucene.search.grouping.function...
  [javadoc] Loading source files for package org.apache.lucene.search.grouping.term...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/grouping/help-doc.html...
  [javadoc] 1 warning

[...truncated 27 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.search.highlight...
  [javadoc] Loading source files for package org.apache.lucene.search.postingshighlight...
  [javadoc] Loading source files for package org.apache.lucene.search.vectorhighlight...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/highlighter/help-doc.html...
  [javadoc] 1 warning

[...truncated 28 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.search.join...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/join/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 27 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.index.memory...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/memory/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 25 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.document...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.index...
  [javadoc] Loading source files for package org.apache.lucene.index.sorter...
  [javadoc] Loading source files for package org.apache.lucene.misc...
  [javadoc] Loading source files for package org.apache.lucene.store...
  [javadoc] Loading source files for package org.apache.lucene.util.fst...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/misc/org/apache/lucene/misc/package-summary.html...
  [javadoc] Copying file /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/misc/src/java/org/apache/lucene/misc/doc-files/ss.baselineTf.svg to directory /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/misc/org/apache/lucene/misc/doc-files...
  [javadoc] Copying file /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/misc/src/java/org/apache/lucene/misc/doc-files/ss.computeLengthNorm.svg to directory /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/misc/org/apache/lucene/misc/doc-files...
  [javadoc] Copying file /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/misc/src/java/org/apache/lucene/misc/doc-files/ss.gnuplot to directory /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/misc/org/apache/lucene/misc/doc-files...
  [javadoc] Copying file /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/misc/src/java/org/apache/lucene/misc/doc-files/ss.hyperbolicTf.svg to directory /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/misc/org/apache/lucene/misc/doc-files...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/misc/help-doc.html...
  [javadoc] 1 warning

[...truncated 24 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.queries...
  [javadoc] Loading source files for package org.apache.lucene.queries.function...
  [javadoc] Loading source files for package org.apache.lucene.queries.function.docvalues...
  [javadoc] Loading source files for package org.apache.lucene.queries.function.valuesource...
  [javadoc] Loading source files for package org.apache.lucene.queries.mlt...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/queries/help-doc.html...
  [javadoc] 1 warning

[...truncated 31 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.queryparser.analyzing...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.queryparser.classic...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.complexPhrase...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.ext...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.builders...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.config...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.messages...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.nodes...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.parser...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.processors...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.core.util...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.messages...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.precedence...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.precedence.processors...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.builders...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.config...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.nodes...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.parser...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.flexible.standard.processors...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.simple...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.surround.parser...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.surround.query...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.xml...
  [javadoc] Loading source files for package org.apache.lucene.queryparser.xml.builders...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/queryparser/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 29 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.replicator...
  [javadoc] Loading source files for package org.apache.lucene.replicator.http...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/replicator/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 24 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.sandbox.queries...
  [javadoc] Loading source files for package org.apache.lucene.sandbox.queries.regex...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/sandbox/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 27 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.spatial...
  [javadoc] Loading source files for package org.apache.lucene.spatial.prefix...
  [javadoc] Loading source files for package org.apache.lucene.spatial.prefix.tree...
  [javadoc] Loading source files for package org.apache.lucene.spatial.query...
  [javadoc] Loading source files for package org.apache.lucene.spatial.util...
  [javadoc] Loading source files for package org.apache.lucene.spatial.vector...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/spatial/help-doc.html...
  [javadoc] 1 warning

[...truncated 32 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.search.spell...
  [javadoc] Loading source files for package org.apache.lucene.search.suggest...
  [javadoc] Loading source files for package org.apache.lucene.search.suggest.analyzing...
  [javadoc] Loading source files for package org.apache.lucene.search.suggest.fst...
  [javadoc] Loading source files for package org.apache.lucene.search.suggest.jaspell...
  [javadoc] Loading source files for package org.apache.lucene.search.suggest.tst...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/suggest/help-doc.html...
  [javadoc] 1 warning

[...truncated 18 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene.analysis...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.lucene.codecs...
  [javadoc] Loading source files for package org.apache.lucene.codecs.appending...
  [javadoc] Loading source files for package org.apache.lucene.codecs.asserting...
  [javadoc] Loading source files for package org.apache.lucene.codecs.bloom...
  [javadoc] Loading source files for package org.apache.lucene.codecs.cheapbastard...
  [javadoc] Loading source files for package org.apache.lucene.codecs.compressing...
  [javadoc] Loading source files for package org.apache.lucene.codecs.compressing.dummy...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene3x...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene40...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene41...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene41ords...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene42...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene45...
  [javadoc] Loading source files for package org.apache.lucene.codecs.mockintblock...
  [javadoc] Loading source files for package org.apache.lucene.codecs.mockrandom...
  [javadoc] Loading source files for package org.apache.lucene.codecs.mocksep...
  [javadoc] Loading source files for package org.apache.lucene.codecs.nestedpulsing...
  [javadoc] Loading source files for package org.apache.lucene.codecs.ramonly...
  [javadoc] Loading source files for package org.apache.lucene.index...
  [javadoc] Loading source files for package org.apache.lucene.search...
  [javadoc] Loading source files for package org.apache.lucene.store...
  [javadoc] Loading source files for package org.apache.lucene.util...
  [javadoc] Loading source files for package org.apache.lucene.util.automaton...
  [javadoc] Loading source files for package org.apache.lucene.util.fst...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/test-framework/help-doc.html...
  [javadoc] 1 warning

  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/docs/test-framework/help-doc.html...
  [javadoc] 1 warning

[...truncated 8547 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.client.solrj...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.client.solrj.beans...
  [javadoc] Loading source files for package org.apache.solr.client.solrj.impl...
  [javadoc] Loading source files for package org.apache.solr.client.solrj.request...
  [javadoc] Loading source files for package org.apache.solr.client.solrj.response...
  [javadoc] Loading source files for package org.apache.solr.client.solrj.util...
  [javadoc] Loading source files for package org.apache.solr.common...
  [javadoc] Loading source files for package org.apache.solr.common.cloud...
  [javadoc] Loading source files for package org.apache.solr.common.luke...
  [javadoc] Loading source files for package org.apache.solr.common.params...
  [javadoc] Loading source files for package org.apache.solr.common.util...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-solrj/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 7 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr...
  [javadoc] Loading source files for package org.apache.solr.analysis...
  [javadoc] Loading source files for package org.apache.solr.client.solrj.embedded...
  [javadoc] Loading source files for package org.apache.solr.cloud...
  [javadoc] Loading source files for package org.apache.solr.common...
  [javadoc] Loading source files for package org.apache.solr.core...
  [javadoc] Loading source files for package org.apache.solr.handler...
  [javadoc] Loading source files for package org.apache.solr.handler.admin...
  [javadoc] Loading source files for package org.apache.solr.handler.component...
  [javadoc] Loading source files for package org.apache.solr.handler.loader...
  [javadoc] Loading source files for package org.apache.solr.highlight...
  [javadoc] Loading source files for package org.apache.solr.internal.csv...
  [javadoc] Loading source files for package org.apache.solr.internal.csv.writer...
  [javadoc] Loading source files for package org.apache.solr.logging...
  [javadoc] Loading source files for package org.apache.solr.logging.jul...
  [javadoc] Loading source files for package org.apache.solr.logging.log4j...
  [javadoc] Loading source files for package org.apache.solr.parser...
  [javadoc] Loading source files for package org.apache.solr.request...
  [javadoc] Loading source files for package org.apache.solr.response...
  [javadoc] Loading source files for package org.apache.solr.response.transform...
  [javadoc] Loading source files for package org.apache.solr.rest...
  [javadoc] Loading source files for package org.apache.solr.rest.schema...
  [javadoc] Loading source files for package org.apache.solr.schema...
  [javadoc] Loading source files for package org.apache.solr.search...
  [javadoc] Loading source files for package org.apache.solr.search.function...
  [javadoc] Loading source files for package org.apache.solr.search.function.distance...
  [javadoc] Loading source files for package org.apache.solr.search.grouping...
  [javadoc] Loading source files for package org.apache.solr.search.grouping.collector...
  [javadoc] Loading source files for package org.apache.solr.search.grouping.distributed...
  [javadoc] Loading source files for package org.apache.solr.search.grouping.distributed.command...
  [javadoc] Loading source files for package org.apache.solr.search.grouping.distributed.requestfactory...
  [javadoc] Loading source files for package org.apache.solr.search.grouping.distributed.responseprocessor...
  [javadoc] Loading source files for package org.apache.solr.search.grouping.distributed.shardresultserializer...
  [javadoc] Loading source files for package org.apache.solr.search.grouping.endresulttransformer...
  [javadoc] Loading source files for package org.apache.solr.search.join...
  [javadoc] Loading source files for package org.apache.solr.search.similarities...
  [javadoc] Loading source files for package org.apache.solr.servlet...
  [javadoc] Loading source files for package org.apache.solr.servlet.cache...
  [javadoc] Loading source files for package org.apache.solr.spelling...
  [javadoc] Loading source files for package org.apache.solr.spelling.suggest...
  [javadoc] Loading source files for package org.apache.solr.spelling.suggest.fst...
  [javadoc] Loading source files for package org.apache.solr.spelling.suggest.jaspell...
  [javadoc] Loading source files for package org.apache.solr.spelling.suggest.tst...
  [javadoc] Loading source files for package org.apache.solr.store.blockcache...
  [javadoc] Loading source files for package org.apache.solr.store.hdfs...
  [javadoc] Loading source files for package org.apache.solr.update...
  [javadoc] Loading source files for package org.apache.solr.update.processor...
  [javadoc] Loading source files for package org.apache.solr.util...
  [javadoc] Loading source files for package org.apache.solr.util.plugin...
  [javadoc] Loading source files for package org.apache.solr.util.stats...
  [javadoc] Loading source files for package org.apache.solr.util.xslt...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-core/org/apache/solr/util/package-summary.html...
  [javadoc] Copying file /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/java/org/apache/solr/util/doc-files/min-should-match.html to directory /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-core/org/apache/solr/util/doc-files...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-core/help-doc.html...
  [javadoc] 1 warning

[...truncated 85 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.client.solrj...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.client.solrj.beans...
  [javadoc] Loading source files for package org.apache.solr.client.solrj.impl...
  [javadoc] Loading source files for package org.apache.solr.client.solrj.request...
  [javadoc] Loading source files for package org.apache.solr.client.solrj.response...
  [javadoc] Loading source files for package org.apache.solr.client.solrj.util...
  [javadoc] Loading source files for package org.apache.solr.common...
  [javadoc] Loading source files for package org.apache.solr.common.cloud...
  [javadoc] Loading source files for package org.apache.solr.common.luke...
  [javadoc] Loading source files for package org.apache.solr.common.params...
  [javadoc] Loading source files for package org.apache.solr.common.util...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-solrj/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 107 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr...
  [javadoc] Loading source files for package org.apache.solr.analysis...
  [javadoc] Loading source files for package org.apache.solr.cloud...
  [javadoc] Loading source files for package org.apache.solr.core...
  [javadoc] Loading source files for package org.apache.solr.update.processor...
  [javadoc] Loading source files for package org.apache.solr.util...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-test-framework/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.experimental
  [javadoc] 1 warning

[...truncated 134 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.schema...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-analysis-extras/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 130 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.handler.clustering...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.handler.clustering.carrot2...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-clustering/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 316 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.handler.dataimport...
  [javadoc] Loading source files for package org.apache.solr.handler.dataimport.config...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-dataimporthandler/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 7 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.handler.dataimport...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-dataimporthandler-extras/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 130 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.handler.dataimport...
  [javadoc] Loading source files for package org.apache.solr.handler.dataimport.config...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-dataimporthandler/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 130 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.handler.extraction...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 139 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.update.processor...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-langid/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal
  [javadoc] 1 warning

[...truncated 1053 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.handler.extraction...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 259 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.handler.extraction...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 7 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.morphlines.solr...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-morphlines-core/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 534 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.handler.extraction...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 259 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.handler.extraction...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 6 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.morphlines.solr...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-morphlines-core/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 7 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.morphlines.cell...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-morphlines-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 7 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.hadoop...
  [javadoc] Loading source files for package org.apache.solr.hadoop.dedup...
  [javadoc] Loading source files for package org.apache.solr.hadoop.morphline...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-map-reduce/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 701 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.handler.extraction...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 259 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.handler.extraction...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 6 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.morphlines.solr...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-morphlines-core/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 6 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.morphlines.cell...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-morphlines-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 314 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Loading source files for package org.apache.solr.handler.extraction...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-cell/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 6 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.morphlines.solr...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-morphlines-core/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 133 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.uima.processor...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-uima/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 130 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.solr.response...
  [javadoc] warning: [options] bootstrap class path not set in conjunction with -source 1.6
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.7.0_51
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/docs/solr-velocity/help-doc.html...
  [javadoc] Note: Custom tags that were not seen:  @lucene.internal, @lucene.experimental
  [javadoc] 1 warning

[...truncated 396 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:398: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:87: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:187: Source checkout is dirty after running tests!!! Offending files:
* ./solr/licenses/jackson-core-asl-1.7.4.jar.sha1
* ./solr/licenses/jackson-mapper-asl-1.7.4.jar.sha1
* ./solr/licenses/jersey-core-1.16.jar.sha1

Total time: 126 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.6.0) - Build # 1206 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1206/
Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.MigrateRouteKeyTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.MigrateRouteKeyTest:     1) Thread[id=9153, name=Timer-26, state=WAITING, group=TGRP-MigrateRouteKeyTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:485)         at java.util.TimerThread.mainLoop(Timer.java:483)         at java.util.TimerThread.run(Timer.java:462)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.MigrateRouteKeyTest: 
   1) Thread[id=9153, name=Timer-26, state=WAITING, group=TGRP-MigrateRouteKeyTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        at java.util.TimerThread.run(Timer.java:462)
	at __randomizedtesting.SeedInfo.seed([3CF7DDAF5DE3E626]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.MigrateRouteKeyTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=9153, name=Timer-26, state=WAITING, group=TGRP-MigrateRouteKeyTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:485)         at java.util.TimerThread.mainLoop(Timer.java:483)         at java.util.TimerThread.run(Timer.java:462)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=9153, name=Timer-26, state=WAITING, group=TGRP-MigrateRouteKeyTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        at java.util.TimerThread.run(Timer.java:462)
	at __randomizedtesting.SeedInfo.seed([3CF7DDAF5DE3E626]:0)


FAILED:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
could not set the new overseer

Stack Trace:
java.lang.AssertionError: could not set the new overseer
	at __randomizedtesting.SeedInfo.seed([3CF7DDAF5DE3E626:BD1153B72ABC861A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.OverseerRolesTest.addOverseerRole2ExistingNodes(OverseerRolesTest.java:120)
	at org.apache.solr.cloud.OverseerRolesTest.doTest(OverseerRolesTest.java:86)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:695)




Build Log:
[...truncated 9556 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 1151295 T3769 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1151304 T3769 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1389983419545
   [junit4]   2> 1151312 T3769 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1151314 T3770 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1151415 T3769 oasc.ZkTestServer.run start zk server on port:53516
   [junit4]   2> 1151417 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1151425 T3776 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c93be91 name:ZooKeeperConnection Watcher:127.0.0.1:53516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1151426 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1151426 T3769 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1151442 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1151446 T3778 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@beed7a0 name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1151447 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1151447 T3769 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1151460 T3769 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1151471 T3769 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1151479 T3769 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1151491 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1151492 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1151507 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1151508 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1151520 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1151521 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1151533 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1151533 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1151545 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1151546 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1151557 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1151558 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1151569 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1151569 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1151581 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1151581 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1151592 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1151593 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1151605 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1151606 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1151617 T3769 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1151630 T3769 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1152031 T3769 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1152040 T3769 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53519
   [junit4]   2> 1152043 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1152043 T3769 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1152044 T3769 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885
   [junit4]   2> 1152045 T3769 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885/'
   [junit4]   2> 1152085 T3769 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885/solr.xml
   [junit4]   2> 1152155 T3769 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1152157 T3769 oasc.CoreContainer.<init> New CoreContainer 2130067861
   [junit4]   2> 1152158 T3769 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885/]
   [junit4]   2> 1152160 T3769 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1152160 T3769 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1152161 T3769 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1152162 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1152162 T3769 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1152163 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1152163 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1152164 T3769 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1152165 T3769 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1152195 T3769 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1152196 T3769 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1152196 T3769 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1152197 T3769 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53516/solr
   [junit4]   2> 1152198 T3769 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1152199 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1152205 T3789 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@594bde51 name:ZooKeeperConnection Watcher:127.0.0.1:53516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1152207 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1152213 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1152218 T3791 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4637ae92 name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1152219 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1152224 T3769 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1152238 T3769 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1152251 T3769 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1152259 T3769 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53519_
   [junit4]   2> 1152262 T3769 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53519_
   [junit4]   2> 1152275 T3769 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1152285 T3769 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1152302 T3769 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1152312 T3769 oasc.Overseer.start Overseer (id=91093953384284163-127.0.0.1:53519_-n_0000000000) starting
   [junit4]   2> 1152326 T3769 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1152343 T3793 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1152346 T3793 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1152348 T3769 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1152358 T3769 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1152379 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1152393 T3792 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1152396 T3769 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885
   [junit4]   2> 1152527 T3769 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885/conf/
   [junit4]   2> 1152535 T3769 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1152536 T3769 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1152536 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1152539 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1152544 T3796 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@124f8f49 name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1152544 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1152549 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1152559 T3769 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1152946 T3769 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1152952 T3769 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53523
   [junit4]   2> 1152956 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1152957 T3769 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1152957 T3769 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389983420799
   [junit4]   2> 1152958 T3769 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1389983420799/'
   [junit4]   2> 1153000 T3769 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389983420799/solr.xml
   [junit4]   2> 1153073 T3769 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1153074 T3769 oasc.CoreContainer.<init> New CoreContainer 642827209
   [junit4]   2> 1153075 T3769 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389983420799/]
   [junit4]   2> 1153076 T3769 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1153077 T3769 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1153078 T3769 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1153078 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1153079 T3769 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1153079 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1153080 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1153081 T3769 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1153081 T3769 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1153125 T3769 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1153126 T3769 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1153127 T3769 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1153127 T3769 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53516/solr
   [junit4]   2> 1153128 T3769 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1153130 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1153136 T3807 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fa38f6d name:ZooKeeperConnection Watcher:127.0.0.1:53516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1153137 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1153143 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1153148 T3809 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61f6f2fd name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1153149 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1153164 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1154179 T3769 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53523_
   [junit4]   2> 1154182 T3769 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53523_
   [junit4]   2> 1154195 T3809 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1154196 T3796 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1154197 T3791 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1154216 T3769 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389983420799
   [junit4]   2> 1154348 T3769 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389983420799/conf/
   [junit4]   2> 1154355 T3769 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1154356 T3769 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1154357 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1154747 T3769 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1154752 T3769 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53526
   [junit4]   2> 1154756 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1154757 T3769 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1154757 T3769 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598
   [junit4]   2> 1154758 T3769 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598/'
   [junit4]   2> 1154800 T3769 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598/solr.xml
   [junit4]   2> 1154885 T3769 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1154886 T3769 oasc.CoreContainer.<init> New CoreContainer 43988416
   [junit4]   2> 1154886 T3769 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598/]
   [junit4]   2> 1154888 T3769 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1154888 T3769 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1154889 T3769 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1154889 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1154891 T3769 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1154891 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1154892 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1154892 T3769 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1154893 T3769 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1154923 T3769 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1154924 T3769 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1154925 T3769 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1154925 T3769 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53516/solr
   [junit4]   2> 1154927 T3769 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1154928 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1154933 T3821 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2281d198 name:ZooKeeperConnection Watcher:127.0.0.1:53516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1154935 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1154942 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1154947 T3823 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75f35310 name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1154948 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1154964 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1155977 T3769 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53526_
   [junit4]   2> 1155981 T3769 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53526_
   [junit4]   2> 1155994 T3809 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1155994 T3796 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1155997 T3791 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1155997 T3823 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1156016 T3769 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598
   [junit4]   2> 1156149 T3769 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598/conf/
   [junit4]   2> 1156155 T3769 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1156156 T3769 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1156157 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1156608 T3769 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1156712 T3769 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53529
   [junit4]   2> 1156712 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1156713 T3769 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1156714 T3769 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389983424398
   [junit4]   2> 1156714 T3769 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389983424398/'
   [junit4]   2> 1156780 T3769 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389983424398/solr.xml
   [junit4]   2> 1156877 T3769 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1156878 T3769 oasc.CoreContainer.<init> New CoreContainer 1839261528
   [junit4]   2> 1156878 T3769 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389983424398/]
   [junit4]   2> 1156881 T3769 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1156881 T3769 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1156882 T3769 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1156883 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1156883 T3769 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1156884 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1156884 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1156885 T3769 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1156885 T3769 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1156921 T3769 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1156922 T3769 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1156924 T3769 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1156924 T3769 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53516/solr
   [junit4]   2> 1156925 T3769 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1156927 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1156933 T3835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5eb47aa2 name:ZooKeeperConnection Watcher:127.0.0.1:53516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1156934 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1156945 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1156950 T3837 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4808f29 name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1156950 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1156966 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1157980 T3769 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53529_
   [junit4]   2> 1157983 T3769 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53529_
   [junit4]   2> 1157994 T3823 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1157997 T3796 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1157998 T3791 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1157999 T3809 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1158001 T3837 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1158020 T3769 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389983424398
   [junit4]   2> 1158213 T3769 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389983424398/conf/
   [junit4]   2> 1158223 T3769 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1158224 T3769 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1158225 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1158606 T3769 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1158612 T3769 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53532
   [junit4]   2> 1158617 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1158618 T3769 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1158619 T3769 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389983426466
   [junit4]   2> 1158619 T3769 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1389983426466/'
   [junit4]   2> 1158674 T3769 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389983426466/solr.xml
   [junit4]   2> 1158742 T3769 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1158743 T3769 oasc.CoreContainer.<init> New CoreContainer 2052060120
   [junit4]   2> 1158744 T3769 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389983426466/]
   [junit4]   2> 1158745 T3769 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1158746 T3769 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1158746 T3769 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1158747 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1158747 T3769 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1158748 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1158749 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1158749 T3769 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1158750 T3769 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1158784 T3769 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1158785 T3769 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1158785 T3769 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1158786 T3769 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53516/solr
   [junit4]   2> 1158787 T3769 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1158788 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1158793 T3849 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@171228a4 name:ZooKeeperConnection Watcher:127.0.0.1:53516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1158795 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1158802 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1158806 T3851 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a8ec738 name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1158806 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1158822 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1159836 T3769 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53532_
   [junit4]   2> 1159840 T3769 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53532_
   [junit4]   2> 1159850 T3837 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1159852 T3823 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1159857 T3791 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1159857 T3851 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1159860 T3796 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1159861 T3809 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1159879 T3769 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389983426466
   [junit4]   2> 1160007 T3769 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389983426466/conf/
   [junit4]   2> 1160013 T3769 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1160015 T3769 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1160015 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1160397 T3769 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1160403 T3769 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53535
   [junit4]   2> 1160407 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1160407 T3769 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1160408 T3769 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389983428256
   [junit4]   2> 1160409 T3769 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389983428256/'
   [junit4]   2> 1160464 T3769 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389983428256/solr.xml
   [junit4]   2> 1160534 T3769 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1160535 T3769 oasc.CoreContainer.<init> New CoreContainer 1577888724
   [junit4]   2> 1160535 T3769 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389983428256/]
   [junit4]   2> 1160537 T3769 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1160538 T3769 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1160538 T3769 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1160539 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1160539 T3769 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1160540 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1160542 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1160542 T3769 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1160542 T3769 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1160574 T3769 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1160575 T3769 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1160576 T3769 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1160577 T3769 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53516/solr
   [junit4]   2> 1160577 T3769 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1160579 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1160585 T3863 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11e51b89 name:ZooKeeperConnection Watcher:127.0.0.1:53516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1160586 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1160592 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1160598 T3865 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@743749e8 name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1160598 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1160614 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1161631 T3769 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53535_
   [junit4]   2> 1161635 T3769 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53535_
   [junit4]   2> 1161648 T3796 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1161649 T3837 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1161654 T3851 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1161656 T3809 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1161656 T3823 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1161657 T3865 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1161658 T3791 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1161677 T3769 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389983428256
   [junit4]   2> 1161874 T3769 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389983428256/conf/
   [junit4]   2> 1161884 T3769 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1161885 T3769 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1161886 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1162285 T3769 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1162294 T3769 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53538
   [junit4]   2> 1162296 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1162297 T3769 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1162297 T3769 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389983430127
   [junit4]   2> 1162298 T3769 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389983430127/'
   [junit4]   2> 1162340 T3769 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389983430127/solr.xml
   [junit4]   2> 1162415 T3769 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1162416 T3769 oasc.CoreContainer.<init> New CoreContainer 1245928776
   [junit4]   2> 1162416 T3769 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389983430127/]
   [junit4]   2> 1162418 T3769 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1162418 T3769 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1162420 T3769 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1162421 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1162422 T3769 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1162422 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1162423 T3769 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1162424 T3769 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1162424 T3769 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1162455 T3769 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1162455 T3769 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1162456 T3769 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1162456 T3769 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53516/solr
   [junit4]   2> 1162457 T3769 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1162459 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1162465 T3877 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ca1480f name:ZooKeeperConnection Watcher:127.0.0.1:53516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1162466 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1162472 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1162477 T3879 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d0892 name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1162477 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1162493 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1163506 T3769 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53538_
   [junit4]   2> 1163510 T3769 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53538_
   [junit4]   2> 1163522 T3837 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1163532 T3823 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1163533 T3796 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1163534 T3791 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1163535 T3865 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1163535 T3809 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1163535 T3851 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1163536 T3879 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1163558 T3769 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389983430127
   [junit4]   2> 1163754 T3769 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389983430127/conf/
   [junit4]   2> 1163764 T3769 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1163766 T3769 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1163766 T3769 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1163768 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1163773 T3882 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cfc86b name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1163774 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1163778 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1163790 T3769 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1163795 T3884 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fa49151 name:ZooKeeperConnection Watcher:127.0.0.1:53516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1163795 T3769 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1163800 T3769 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1163813 T3858 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=4&maxShardsPerNode=2&name=testOverseerCol&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 1163817 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1163826 T3793 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1163826 T3793 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1163829 T3793 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1163830 T3793 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 1163830 T3793 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 1163847 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1163853 T3792 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1163855 T3792 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 1163856 T3792 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 1163883 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1163888 T3882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163889 T3837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163889 T3884 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163889 T3823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163890 T3791 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163890 T3851 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163892 T3865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163893 T3879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163895 T3809 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163895 T3796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1163949 T3793 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 1163949 T3793 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:53526_
   [junit4]   2> 1163950 T3793 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:53532_
   [junit4]   2> 1163951 T3793 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:53535_
   [junit4]   2> 1163959 T3793 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:53519_
   [junit4]   2> 1163966 T3793 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:53529_
   [junit4]   2> 1163970 T3844 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 1163970 T3817 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 1163970 T3793 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:53523_
   [junit4]   2> 1163975 T3859 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 1163976 T3833 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 1163973 T3786 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 1163979 T3793 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:53538_
   [junit4]   2> 1163981 T3817 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1163982 T3844 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1163988 T3802 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 1163987 T3859 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1163986 T3833 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1163983 T3793 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:53526_
   [junit4]   2> 1163991 T3873 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 1163995 T3802 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1163997 T3873 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1163998 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164002 T3786 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1164002 T3819 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 1164006 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164007 T3819 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1164011 T3792 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1164013 T3792 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53526",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53526_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1164027 T3792 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53532",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53532_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1164036 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164040 T3792 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53535",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53535_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1164049 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164055 T3792 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53529",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53529_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1164065 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164071 T3792 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53523",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53523_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1164082 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164087 T3792 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53538",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53538_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1164097 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164102 T3792 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53519",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53519_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1164112 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164116 T3792 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53526",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53526_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1164126 T3791 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1164240 T3837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164240 T3791 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164242 T3796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164242 T3884 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164242 T3882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164246 T3865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164247 T3851 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164247 T3823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164248 T3809 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164248 T3879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1164983 T3817 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 1164985 T3817 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598/testOverseerCol_shard1_replica1
   [junit4]   2> 1164985 T3817 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1164988 T3817 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1164988 T3817 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1164991 T3844 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 1164992 T3859 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 1164993 T3844 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389983426466/testOverseerCol_shard1_replica2
   [junit4]   2> 1164994 T3844 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1164994 T3833 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 1164995 T3859 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389983428256/testOverseerCol_shard2_replica1
   [junit4]   2> 1164997 T3859 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1164997 T3802 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 1164998 T3833 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389983424398/testOverseerCol_shard3_replica1
   [junit4]   2> 1164999 T3833 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1165000 T3802 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389983420799/testOverseerCol_shard3_replica2
   [junit4]   2> 1165000 T3802 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1165001 T3873 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 1165003 T3833 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1165003 T3833 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1165004 T3873 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389983430127/testOverseerCol_shard4_replica1
   [junit4]   2> 1165004 T3873 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1165006 T3859 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1165006 T3859 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1165007 T3786 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 1165007 T3844 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1165009 T3844 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1165010 T3819 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 1165012 T3819 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598/testOverseerCol_shard4_replica2
   [junit4]   2> 1165013 T3819 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1165009 T3817 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598/testOverseerCol_shard1_replica1/'
   [junit4]   2> 1165015 T3833 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389983424398/testOverseerCol_shard3_replica1/'
   [junit4]   2> 1165019 T3844 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1389983426466/testOverseerCol_shard1_replica2/'
   [junit4]   2> 1165009 T3786 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885/testOverseerCol_shard2_replica2
   [junit4]   2> 1165020 T3786 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1165027 T3819 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1165027 T3819 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1165029 T3802 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1165030 T3802 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1165044 T3786 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1165044 T3786 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1165054 T3873 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1165055 T3873 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1165055 T3859 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389983428256/testOverseerCol_shard2_replica1/'
   [junit4]   2> 1165059 T3802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1389983420799/testOverseerCol_shard3_replica2/'
   [junit4]   2> 1165064 T3819 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1389983422598/testOverseerCol_shard4_replica2/'
   [junit4]   2> 1165083 T3786 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885/testOverseerCol_shard2_replica2/'
   [junit4]   2> 1165101 T3873 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389983430127/testOverseerCol_shard4_replica1/'
   [junit4]   2> 1165415 T3786 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1165418 T3844 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1165454 T3819 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1165486 T3817 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1165494 T3859 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1165499 T3833 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1165506 T3873 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1165533 T3802 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1165712 T3844 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1165719 T3844 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1165810 T3844 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 1165814 T3786 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1165835 T3786 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1165879 T3786 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 1165884 T3802 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1165908 T3817 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1165917 T3859 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1165922 T3802 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1165935 T3817 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1165943 T3859 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1165981 T3802 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 1165981 T3817 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 1165988 T3833 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1165989 T3819 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1166000 T3833 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1166004 T3859 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 1166006 T3819 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1166050 T3819 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 1166055 T3873 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1166089 T3873 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1166103 T3873 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 1166108 T3833 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 1169531 T3786 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169537 T3786 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169538 T3786 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1169545 T3802 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169562 T3802 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169563 T3802 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1169584 T3786 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1169589 T3786 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1169595 T3802 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1169605 T3802 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1169624 T3817 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169625 T3817 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169626 T3817 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1169658 T3817 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1169669 T3844 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169670 T3844 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169671 T3844 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1169692 T3844 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1169722 T3819 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169723 T3819 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169724 T3819 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1169733 T3844 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1169750 T3819 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1169762 T3819 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1169791 T3817 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1169806 T3833 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169808 T3833 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169810 T3833 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1169824 T3833 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1169827 T3833 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1169959 T3873 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169960 T3873 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169963 T3873 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1169960 T3859 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169970 T3859 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1169976 T3873 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1169980 T3859 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1169981 T3873 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1170011 T3859 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1170016 T3859 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1170026 T3844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1170034 T3786 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1170053 T3844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1170055 T3786 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1170057 T3802 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1170075 T3786 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1170080 T3802 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1170091 T3844 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1170093 T3786 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1170094 T3786 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1170093 T3802 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1170095 T3786 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1170111 T3786 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1170112 T3786 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1170114 T3786 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1170115 T3786 oasc.SolrCore.<init> [testOverseerCol_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885/testOverseerCol_shard2_replica2/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389983419885/testOverseerCol_shard2_replica2/dat

[...truncated too long message...]

t4]   2> 	        at java.util.TimerThread.run(Timer.java:462)
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=vi_VN, timezone=Europe/Monaco
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Apple Inc. 1.6.0_65 (64-bit)/cpus=2,threads=2,free=44612176,total=532545536
   [junit4]   2> NOTE: All tests run in this JVM: [PingRequestHandlerTest, MultiTermTest, ClusterStateUpdateTest, OverseerCollectionProcessorTest, ConnectionManagerTest, SpellCheckCollatorTest, HdfsChaosMonkeySafeLeaderTest, SuggesterWFSTTest, CopyFieldTest, InfoHandlerTest, JsonLoaderTest, SyncSliceTest, TestFastOutputStream, BasicZkTest, TestCloudManagedSchemaAddField, CircularListTest, TestSearchPerf, TestDocumentBuilder, TestRealTimeGet, TestCursorMarkWithoutUniqueKey, TestRandomMergePolicy, TestDFRSimilarityFactory, LeaderElectionTest, TestUtils, TestStressReorder, OverseerTest, TestLFUCache, EchoParamsTest, SolrXmlInZkTest, TestComponentsName, TestZkChroot, LoggingHandlerTest, TestSolrXMLSerializer, ChaosMonkeySafeLeaderTest, TestSuggestSpellingConverter, TestSurroundQueryParser, TestStressRecovery, DocValuesMultiTest, ScriptEngineTest, TestQueryUtils, BJQParserTest, XmlUpdateRequestHandlerTest, UnloadDistributedZkTest, DirectSolrConnectionTest, TestArbitraryIndexDir, HardAutoCommitTest, StressHdfsTest, ConvertedLegacyTest, TestAtomicUpdateErrorCases, TestLMJelinekMercerSimilarityFactory, CoreAdminHandlerTest, TestCharFilters, DateMathParserTest, DistributedTermsComponentTest, TestRTGBase, QueryParsingTest, DOMUtilTest, PluginInfoTest, PreAnalyzedUpdateProcessorTest, TestRandomFaceting, AlternateDirectoryTest, HdfsUnloadDistributedZkTest, TestNonNRTOpen, SliceStateUpdateTest, CacheHeaderTest, TestCodecSupport, HdfsCollectionsAPIDistributedZkTest, AddBlockUpdateTest, TestSchemaSimilarityResource, HdfsBasicDistributedZkTest, SortByFunctionTest, PrimitiveFieldTypeTest, TestSolrXml, ShardRoutingCustomTest, StatelessScriptUpdateProcessorFactoryTest, TestImplicitCoreProperties, TestPseudoReturnFields, DebugComponentTest, FileBasedSpellCheckerTest, TestReplicationHandler, TestFastLRUCache, CurrencyFieldOpenExchangeTest, ResponseLogComponentTest, SchemaVersionSpecificBehaviorTest, WordBreakSolrSpellCheckerTest, DocValuesMissingTest, CurrencyFieldXmlFileTest, TermVectorComponentDistributedTest, AnalysisAfterCoreReloadTest, TestCollapseQParserPlugin, TestFreeTextSuggestions, TestIBSimilarityFactory, LeaderElectionIntegrationTest, TestSolrQueryParserDefaultOperatorResource, CursorMarkTest, TestSolrDeletionPolicy2, TestFastWriter, TestDistributedSearch, TestQuerySenderNoQuery, TestRemoteStreaming, TestBM25SimilarityFactory, TestSolrQueryParser, TestWordDelimiterFilterFactory, OverseerRolesTest, BlockDirectoryTest, TestLuceneMatchVersion, TestFiltering, TestCloudManagedSchema, DateFieldTest, DocValuesTest, TestNRTOpen, TestFieldCollectionResource, TestDefaultSearchFieldResource, BinaryUpdateRequestHandlerTest, FieldMutatingUpdateProcessorTest, SuggesterTSTTest, TestReversedWildcardFilterFactory, TestDistributedMissingSort, CoreAdminCreateDiscoverTest, SolrCoreTest, TestPostingsSolrHighlighter, PrimUtilsTest, HighlighterConfigTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestReloadAndDeleteDocs, TestSolrJ, PreAnalyzedFieldTest, HdfsLockFactoryTest, TestSolrXmlPersistence, TestFuzzyAnalyzedSuggestions, DocumentBuilderTest, DisMaxRequestHandlerTest, DeleteShardTest, TestCollationField, TestLMDirichletSimilarityFactory, SoftAutoCommitTest, UUIDFieldTest, ExternalFileFieldSortTest, TermsComponentTest, DocumentAnalysisRequestHandlerTest, TestPartialUpdateDeduplication, TestNumberUtils, DistributedQueryComponentCustomSortTest, DirectUpdateHandlerTest, URLClassifyProcessorTest, CoreContainerCoreInitFailuresTest, BadCopyFieldTest, TestStressUserVersions, TestSolrDeletionPolicy1, TestCoreContainer, SuggestComponentTest, TestPhraseSuggestions, TestDynamicFieldResource, IndexBasedSpellCheckerTest, IndexSchemaTest, MBeansHandlerTest, SignatureUpdateProcessorFactoryTest, SolrCmdDistributorTest, TestStressLucene, SolrTestCaseJ4Test, ZkNodePropsTest, SearchHandlerTest, DistributedSuggestComponentTest, TestLRUCache, SpellCheckComponentTest, TestReload, DistanceFunctionTest, ResourceLoaderTest, SliceStateTest, DirectUpdateHandlerOptimizeTest, BasicDistributedZkTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ZkSolrClientTest, TestRandomDVFaceting, ZkCLITest, AutoCommitTest, TestDistributedGrouping, TestFaceting, TestRecovery, TestHashPartitioner, DistributedSpellCheckComponentTest, ZkControllerTest, TestJoin, TestMultiCoreConfBootstrap, TestStressVersions, TestRangeQuery, SimpleFacetsTest, TestSolr4Spatial, StatsComponentTest, TestGroupingSearch, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, TestSort, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores, TestBadConfig, TestIndexSearcher, HighlighterTest, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, TestExtendedDismaxParser, SuggesterFSTTest, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTest, TestTrie, SpatialFilterTest, TestCSVLoader, PolyFieldTest, NoCacheHeaderTest, SolrCoreCheckLockOnStartupTest, TestUpdate, QueryEqualityTest, DefaultValueUpdateProcessorTest, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, LukeRequestHandlerTest, StandardRequestHandlerTest, TestWriterPerf, DirectSolrSpellCheckerTest, TestQueryTypes, TestOmitPositions, RequestHandlersTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, TestIndexingPerformance, MoreLikeThisHandlerTest, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, SolrPluginUtilsTest, TestJmxIntegration, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, JSONWriterTest, TestCSVResponseWriter, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, TestBinaryResponseWriter, SOLR749Test, TestQuerySenderListener, UpdateParamsTest, TestSolrIndexConfig, SolrIndexConfigTest, BadComponentTest, TestMergePolicyConfig, SampleTest, TestDocSet, TestBinaryField, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestSolrCoreProperties, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, SynonymTokenizerTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, TestDefaultSimilarityFactory, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, TimeZoneUtilsTest, OpenExchangeRatesOrgProviderTest, ChaosMonkeyNothingIsSafeTest, TestSystemIdResolver, SpellingQueryConverterTest, RAMDirectoryFactoryTest, ClusterStateTest, SystemInfoHandlerTest, FileUtilsTest, CursorPagingTest, TestCollationKeyRangeQueries, AliasIntegrationTest, AssignTest, DeleteReplicaTest, DistribCursorPagingTest, MigrateRouteKeyTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=MigrateRouteKeyTest -Dtests.seed=3CF7DDAF5DE3E626 -Dtests.slow=true -Dtests.locale=vi_VN -Dtests.timezone=Europe/Monaco -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s | MigrateRouteKeyTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.MigrateRouteKeyTest: 
   [junit4]    >    1) Thread[id=9153, name=Timer-26, state=WAITING, group=TGRP-MigrateRouteKeyTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:485)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:483)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:462)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3CF7DDAF5DE3E626]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=9153, name=Timer-26, state=WAITING, group=TGRP-MigrateRouteKeyTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:485)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:483)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:462)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3CF7DDAF5DE3E626]:0)
   [junit4] Completed in 82.17s, 1 test, 2 errors <<< FAILURES!

[...truncated 142 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1541 tests, 2 suite-level errors, 1 failure, 202 ignored (5 assumptions)

Total time: 94 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.6.0) - Build # 1205 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1205/
Java: 64bit/jdk1.6.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
could not set the new overseer

Stack Trace:
java.lang.AssertionError: could not set the new overseer
	at __randomizedtesting.SeedInfo.seed([D8E8DAE301AA440F:590E54FB76F52433]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.OverseerRolesTest.addOverseerRole2ExistingNodes(OverseerRolesTest.java:120)
	at org.apache.solr.cloud.OverseerRolesTest.doTest(OverseerRolesTest.java:86)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:695)




Build Log:
[...truncated 10273 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 2643481 T9601 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2643486 T9601 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1389960516438
   [junit4]   2> 2643487 T9601 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2643488 T9602 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2643589 T9601 oasc.ZkTestServer.run start zk server on port:53662
   [junit4]   2> 2643591 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2643600 T9608 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3dbb05e1 name:ZooKeeperConnection Watcher:127.0.0.1:53662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2643601 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2643602 T9601 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2643621 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2643625 T9610 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@601d5ab name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2643626 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2643626 T9601 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2643638 T9601 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2643649 T9601 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2643658 T9601 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2643670 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2643672 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2643688 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2643689 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2643702 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2643704 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2643716 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2643717 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2643728 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2643730 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2643743 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2643745 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2643755 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2643756 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2643768 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2643769 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2643781 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2643782 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2643794 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2643795 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2643810 T9601 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2643811 T9601 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2644122 T9601 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2644134 T9601 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53665
   [junit4]   2> 2644134 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2644135 T9601 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2644136 T9601 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389960516775
   [junit4]   2> 2644137 T9601 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389960516775/'
   [junit4]   2> 2644166 T9601 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389960516775/solr.xml
   [junit4]   2> 2644228 T9601 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2644229 T9601 oasc.CoreContainer.<init> New CoreContainer 898885307
   [junit4]   2> 2644230 T9601 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389960516775/]
   [junit4]   2> 2644231 T9601 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2644232 T9601 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2644232 T9601 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2644233 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2644233 T9601 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2644234 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2644235 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2644235 T9601 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2644236 T9601 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2644257 T9601 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2644258 T9601 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2644259 T9601 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2644260 T9601 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53662/solr
   [junit4]   2> 2644260 T9601 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2644261 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2644268 T9621 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1111a78 name:ZooKeeperConnection Watcher:127.0.0.1:53662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2644269 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2644275 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2644280 T9623 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26f7e187 name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2644281 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2644286 T9601 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2644300 T9601 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2644311 T9601 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2644318 T9601 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53665_
   [junit4]   2> 2644320 T9601 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53665_
   [junit4]   2> 2644333 T9601 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2644341 T9601 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2644357 T9601 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2644365 T9601 oasc.Overseer.start Overseer (id=91092452405739523-127.0.0.1:53665_-n_0000000000) starting
   [junit4]   2> 2644376 T9601 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2644390 T9625 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2644392 T9601 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2644393 T9625 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2644405 T9601 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2644411 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2644425 T9624 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2644428 T9601 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389960516775
   [junit4]   2> 2644569 T9601 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389960516775/conf/
   [junit4]   2> 2644576 T9601 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2644580 T9601 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2644580 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2644582 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2644588 T9628 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12f2c751 name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2644588 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2644593 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2644604 T9601 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2644938 T9601 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2644944 T9601 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53669
   [junit4]   2> 2644946 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2644947 T9601 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2644947 T9601 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389960517556
   [junit4]   2> 2644948 T9601 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389960517556/'
   [junit4]   2> 2644978 T9601 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389960517556/solr.xml
   [junit4]   2> 2645040 T9601 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2645042 T9601 oasc.CoreContainer.<init> New CoreContainer 536623563
   [junit4]   2> 2645042 T9601 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389960517556/]
   [junit4]   2> 2645045 T9601 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2645045 T9601 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2645046 T9601 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2645046 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2645047 T9601 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2645047 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2645048 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2645048 T9601 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2645049 T9601 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2645072 T9601 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2645072 T9601 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2645073 T9601 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2645073 T9601 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53662/solr
   [junit4]   2> 2645075 T9601 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2645076 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2645081 T9639 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68364ed4 name:ZooKeeperConnection Watcher:127.0.0.1:53662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2645082 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2645088 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2645093 T9641 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@510aa09a name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2645094 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2645111 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2646125 T9601 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53669_
   [junit4]   2> 2646128 T9601 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53669_
   [junit4]   2> 2646141 T9623 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2646141 T9628 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2646143 T9641 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2646161 T9601 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389960517556
   [junit4]   2> 2646301 T9601 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389960517556/conf/
   [junit4]   2> 2646310 T9601 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2646312 T9601 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2646312 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2646638 T9601 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2646644 T9601 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53672
   [junit4]   2> 2646646 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2646646 T9601 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2646647 T9601 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389960519264
   [junit4]   2> 2646647 T9601 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389960519264/'
   [junit4]   2> 2646676 T9601 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389960519264/solr.xml
   [junit4]   2> 2646720 T9601 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2646721 T9601 oasc.CoreContainer.<init> New CoreContainer 1914449213
   [junit4]   2> 2646721 T9601 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389960519264/]
   [junit4]   2> 2646723 T9601 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2646723 T9601 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2646724 T9601 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2646724 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2646725 T9601 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2646725 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2646726 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2646727 T9601 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2646727 T9601 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2646753 T9601 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2646755 T9601 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2646755 T9601 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2646755 T9601 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53662/solr
   [junit4]   2> 2646756 T9601 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2646757 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2646762 T9653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17de3a52 name:ZooKeeperConnection Watcher:127.0.0.1:53662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2646764 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2646769 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2646774 T9655 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4618c3fc name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2646774 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2646788 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2647803 T9601 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53672_
   [junit4]   2> 2647806 T9601 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53672_
   [junit4]   2> 2647818 T9623 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2647819 T9641 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2647820 T9628 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2647821 T9655 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2647843 T9601 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389960519264
   [junit4]   2> 2647988 T9601 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389960519264/conf/
   [junit4]   2> 2648000 T9601 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2648002 T9601 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2648002 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2648324 T9601 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2648350 T9601 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53675
   [junit4]   2> 2648351 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2648351 T9601 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2648353 T9601 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389960520954
   [junit4]   2> 2648354 T9601 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389960520954/'
   [junit4]   2> 2648383 T9601 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389960520954/solr.xml
   [junit4]   2> 2648427 T9601 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2648429 T9601 oasc.CoreContainer.<init> New CoreContainer 1148142764
   [junit4]   2> 2648429 T9601 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389960520954/]
   [junit4]   2> 2648431 T9601 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2648431 T9601 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2648432 T9601 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2648432 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2648433 T9601 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2648434 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2648435 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2648436 T9601 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2648437 T9601 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2648461 T9601 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2648462 T9601 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2648463 T9601 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2648463 T9601 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53662/solr
   [junit4]   2> 2648464 T9601 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2648466 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2648471 T9667 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65059b4d name:ZooKeeperConnection Watcher:127.0.0.1:53662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2648471 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2648477 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2648482 T9669 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c7b53a8 name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2648482 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2648497 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2649509 T9601 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53675_
   [junit4]   2> 2649512 T9601 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53675_
   [junit4]   2> 2649526 T9623 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649527 T9628 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649530 T9669 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649531 T9641 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649532 T9655 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649550 T9601 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389960520954
   [junit4]   2> 2649693 T9601 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389960520954/conf/
   [junit4]   2> 2649702 T9601 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2649704 T9601 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2649705 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2650045 T9601 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2650050 T9601 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53678
   [junit4]   2> 2650055 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2650056 T9601 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2650056 T9601 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657
   [junit4]   2> 2650057 T9601 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657/'
   [junit4]   2> 2650086 T9601 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657/solr.xml
   [junit4]   2> 2650148 T9601 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2650149 T9601 oasc.CoreContainer.<init> New CoreContainer 116810615
   [junit4]   2> 2650149 T9601 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657/]
   [junit4]   2> 2650152 T9601 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2650152 T9601 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2650152 T9601 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2650153 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2650154 T9601 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2650154 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2650155 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2650156 T9601 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2650157 T9601 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2650179 T9601 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2650180 T9601 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2650180 T9601 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2650181 T9601 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53662/solr
   [junit4]   2> 2650182 T9601 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2650184 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2650188 T9681 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19fc1d33 name:ZooKeeperConnection Watcher:127.0.0.1:53662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2650189 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2650195 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2650199 T9683 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@776ea64b name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2650200 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2650214 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2651227 T9601 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53678_
   [junit4]   2> 2651230 T9601 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53678_
   [junit4]   2> 2651243 T9641 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2651244 T9655 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2651245 T9669 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2651245 T9628 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2651246 T9623 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2651247 T9683 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2651263 T9601 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657
   [junit4]   2> 2651407 T9601 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657/conf/
   [junit4]   2> 2651415 T9601 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2651417 T9601 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2651417 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2651731 T9601 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2651736 T9601 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53681
   [junit4]   2> 2651739 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2651740 T9601 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2651740 T9601 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389960524370
   [junit4]   2> 2651741 T9601 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389960524370/'
   [junit4]   2> 2651770 T9601 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389960524370/solr.xml
   [junit4]   2> 2651814 T9601 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2651815 T9601 oasc.CoreContainer.<init> New CoreContainer 121191857
   [junit4]   2> 2651815 T9601 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389960524370/]
   [junit4]   2> 2651817 T9601 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2651817 T9601 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2651818 T9601 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2651818 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2651819 T9601 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2651819 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2651820 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2651820 T9601 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2651821 T9601 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2651844 T9601 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2651845 T9601 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2651845 T9601 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2651846 T9601 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53662/solr
   [junit4]   2> 2651846 T9601 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2651848 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2651854 T9695 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f01332b name:ZooKeeperConnection Watcher:127.0.0.1:53662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2651855 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2651861 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2651865 T9697 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7039d31b name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2651866 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2651880 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2652896 T9601 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53681_
   [junit4]   2> 2652899 T9601 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53681_
   [junit4]   2> 2652910 T9655 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652916 T9641 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652917 T9623 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652917 T9683 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652919 T9669 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652920 T9697 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652920 T9628 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652936 T9601 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389960524370
   [junit4]   2> 2653101 T9601 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389960524370/conf/
   [junit4]   2> 2653109 T9601 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2653111 T9601 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2653112 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2653417 T9601 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2653422 T9601 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53684
   [junit4]   2> 2653426 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2653426 T9601 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2653427 T9601 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389960526063
   [junit4]   2> 2653427 T9601 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty6-1389960526063/'
   [junit4]   2> 2653456 T9601 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389960526063/solr.xml
   [junit4]   2> 2653500 T9601 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2653501 T9601 oasc.CoreContainer.<init> New CoreContainer 959334916
   [junit4]   2> 2653502 T9601 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389960526063/]
   [junit4]   2> 2653503 T9601 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2653504 T9601 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2653504 T9601 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2653505 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2653505 T9601 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2653506 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2653507 T9601 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2653507 T9601 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2653508 T9601 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2653529 T9601 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2653530 T9601 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2653531 T9601 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2653532 T9601 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53662/solr
   [junit4]   2> 2653532 T9601 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2653534 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2653538 T9709 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dfd69f0 name:ZooKeeperConnection Watcher:127.0.0.1:53662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2653539 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2653545 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2653549 T9711 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3757d7c4 name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2653550 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2653563 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2654576 T9601 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53684_
   [junit4]   2> 2654580 T9601 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53684_
   [junit4]   2> 2654597 T9628 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654599 T9697 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654602 T9669 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654603 T9655 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654604 T9623 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654604 T9711 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654605 T9641 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654607 T9683 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654621 T9601 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389960526063
   [junit4]   2> 2654719 T9601 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389960526063/conf/
   [junit4]   2> 2654725 T9601 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2654727 T9601 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2654729 T9601 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2654731 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2654735 T9714 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2966c3b6 name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2654736 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2654740 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2654750 T9601 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2654754 T9716 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50bcea00 name:ZooKeeperConnection Watcher:127.0.0.1:53662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2654755 T9601 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2654759 T9601 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2654772 T9615 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=4&maxShardsPerNode=2&name=testOverseerCol&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 2654775 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654778 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654779 T9625 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 2654780 T9625 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 2654782 T9625 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 2654782 T9625 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 2654783 T9625 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 2654798 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654803 T9624 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2654805 T9624 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 2654806 T9624 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 2654814 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654817 T9697 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654817 T9669 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654818 T9716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654818 T9628 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654819 T9711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654818 T9714 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654820 T9641 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654820 T9655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654818 T9623 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654823 T9683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654900 T9625 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 2654901 T9625 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:53678_
   [junit4]   2> 2654902 T9625 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:53669_
   [junit4]   2> 2654903 T9625 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:53681_
   [junit4]   2> 2654910 T9625 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:53684_
   [junit4]   2> 2654915 T9625 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:53672_
   [junit4]   2> 2654916 T9674 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 2654919 T9625 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:53665_
   [junit4]   2> 2654920 T9705 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 2654926 T9647 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 2654927 T9616 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 2654929 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654925 T9674 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654924 T9689 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 2654923 T9625 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:53675_
   [junit4]   2> 2654930 T9705 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654926 T9636 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 2654933 T9647 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654932 T9625 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:53678_
   [junit4]   2> 2654937 T9689 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654938 T9663 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 2654939 T9616 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654942 T9636 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654943 T9675 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 2654944 T9663 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654946 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654946 T9675 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654949 T9624 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2654951 T9624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53678",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53678_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654963 T9624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53684",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53684_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654972 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654976 T9624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53672",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53672_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654984 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654989 T9624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53681",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53681_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654998 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2655003 T9624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53665",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53665_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2655012 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2655017 T9624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53669",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53669_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2655025 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2655030 T9624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53675",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53675_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2655039 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2655046 T9624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53678",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53678_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2655055 T9623 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2655164 T9623 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655165 T9716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655166 T9697 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655165 T9655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655168 T9628 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655165 T9669 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655165 T9714 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655167 T9641 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655166 T9711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655173 T9683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655931 T9674 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 2655933 T9705 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 2655933 T9674 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657/testOverseerCol_shard1_replica1
   [junit4]   2> 2655935 T9674 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655937 T9705 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389960526063/testOverseerCol_shard2_replica2
   [junit4]   2> 2655938 T9705 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655937 T9647 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 2655940 T9689 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 2655942 T9647 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389960519264/testOverseerCol_shard3_replica1
   [junit4]   2> 2655944 T9647 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655944 T9636 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 2655944 T9616 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 2655944 T9674 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655950 T9674 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655951 T9616 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389960516775/testOverseerCol_shard3_replica2
   [junit4]   2> 2655952 T9616 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655944 T9689 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389960524370/testOverseerCol_shard2_replica1
   [junit4]   2> 2655954 T9689 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655951 T9647 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655956 T9647 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655950 T9636 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389960517556/testOverseerCol_shard1_replica2
   [junit4]   2> 2655958 T9636 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655949 T9675 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 2655946 T9663 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 2655963 T9675 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657/testOverseerCol_shard4_replica2
   [junit4]   2> 2655964 T9675 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655946 T9705 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655966 T9705 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655967 T9689 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655968 T9689 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655969 T9636 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655964 T9663 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389960520954/testOverseerCol_shard4_replica1
   [junit4]   2> 2655971 T9663 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655956 T9616 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655972 T9616 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655970 T9636 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655969 T9674 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657/testOverseerCol_shard1_replica1/'
   [junit4]   2> 2655969 T9647 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389960519264/testOverseerCol_shard3_replica1/'
   [junit4]   2> 2655978 T9663 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655978 T9663 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655980 T9705 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty6-1389960526063/testOverseerCol_shard2_replica2/'
   [junit4]   2> 2656018 T9675 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2656019 T9675 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2656025 T9689 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389960524370/testOverseerCol_shard2_replica1/'
   [junit4]   2> 2656035 T9636 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389960517556/testOverseerCol_shard1_replica2/'
   [junit4]   2> 2656037 T9616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389960516775/testOverseerCol_shard3_replica2/'
   [junit4]   2> 2656038 T9663 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389960520954/testOverseerCol_shard4_replica1/'
   [junit4]   2> 2656044 T9675 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389960522657/testOverseerCol_shard4_replica2/'
   [junit4]   2> 2656229 T9674 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2656292 T9636 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2656296 T9689 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2656316 T9616 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2656320 T9663 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2656336 T9675 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2656378 T9705 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2656414 T9647 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2656499 T9674 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656504 T9636 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656528 T9674 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656535 T9636 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656547 T9705 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656552 T9705 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656566 T9674 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 2656578 T9705 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 2656579 T9636 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 2656593 T9647 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656600 T9647 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656635 T9689 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656640 T9647 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 2656648 T9675 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656655 T9616 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656657 T9663 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656660 T9689 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656665 T9675 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656668 T9616 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656665 T9663 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656689 T9689 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 2656691 T9616 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 2656722 T9663 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 2656818 T9675 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 2659301 T9647 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659323 T9647 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659325 T9647 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2659339 T9647 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2659349 T9616 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659351 T9616 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659352 T9616 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2659356 T9647 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2659363 T9616 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2659390 T9616 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2659498 T9705 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659499 T9705 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659500 T9705 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2659519 T9705 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2659520 T9674 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659521 T9674 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659522 T9705 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2659522 T9674 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2659532 T9663 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659533 T9663 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659544 T9689 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659556 T9689 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659559 T9689 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2659546 T9674 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2659554 T9663 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2659567 T9674 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2659574 T9663 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2659547 T9636 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659579 T9689 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2659579 T9636 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659582 T9636 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2659588 T9675 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659589 T9675 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2659591 T9675 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2659589 T9636 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2659592 T9689 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2659599 T9663 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2659599 T9675 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2659607 T9636 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2659612 T9647 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2659641 T9675 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2659649 T9616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2659660 T9647 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2659679 T9616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2659681 T9647 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2659698 T9647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2659699 T9647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2659700 T9647 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2659704 T9616 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2659708 T9647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 26

[...truncated too long message...]

ception$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testOverseerCol/leader_elect/shard4/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_PR, timezone=Europe/Monaco
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Apple Inc. 1.6.0_65 (64-bit)/cpus=2,threads=1,free=120906488,total=250224640
   [junit4]   2> NOTE: All tests run in this JVM: [ShardRoutingTest, BJQParserTest, DistributedSpellCheckComponentTest, AnalysisAfterCoreReloadTest, TestBinaryField, DocumentBuilderTest, TestPartialUpdateDeduplication, TestElisionMultitermQuery, TestNumberUtils, TestIndexSearcher, NumericFieldsTest, IndexSchemaTest, HdfsChaosMonkeySafeLeaderTest, TestBinaryResponseWriter, DistributedDebugComponentTest, RAMDirectoryFactoryTest, HdfsCollectionsAPIDistributedZkTest, TestAtomicUpdateErrorCases, TestSolrXml, TestSort, TestShardHandlerFactory, TriLevelCompositeIdRoutingTest, SolrCmdDistributorTest, TestSchemaNameResource, TestDistributedSearch, CacheHeaderTest, TestArbitraryIndexDir, TestSolrDeletionPolicy2, PathHierarchyTokenizerFactoryTest, FastVectorHighlighterTest, OpenCloseCoreStressTest, SimplePostToolTest, TestSolrJ, TestNRTOpen, SampleTest, TestRandomDVFaceting, TermsComponentTest, TestBM25SimilarityFactory, ClusterStateTest, StandardRequestHandlerTest, BadCopyFieldTest, TestAddFieldRealTimeGet, HdfsLockFactoryTest, SuggestComponentTest, TestComponentsName, HdfsBasicDistributedZkTest, CSVRequestHandlerTest, CoreMergeIndexesAdminHandlerTest, StatelessScriptUpdateProcessorFactoryTest, PreAnalyzedUpdateProcessorTest, TestCloudManagedSchemaAddField, SynonymTokenizerTest, TestDocBasedVersionConstraints, MigrateRouteKeyTest, SpellCheckComponentTest, SystemInfoHandlerTest, TestSolrXMLSerializer, IndexBasedSpellCheckerTest, EchoParamsTest, TestDFRSimilarityFactory, SchemaVersionSpecificBehaviorTest, TestSearchPerf, CursorPagingTest, TestPerFieldSimilarity, DistribCursorPagingTest, InfoHandlerTest, TestDynamicFieldCollectionResource, SolrCoreTest, TestNonNRTOpen, ChangedSchemaMergeTest, TestFieldTypeResource, TestAnalyzeInfixSuggestions, TestRemoteStreaming, JSONWriterTest, QueryEqualityTest, TestSimpleQParserPlugin, TestLuceneMatchVersion, StatsComponentTest, TestReplicationHandler, TestQueryTypes, TestReloadAndDeleteDocs, HighlighterConfigTest, FileUtilsTest, TestSolrQueryParserDefaultOperatorResource, AssignTest, SolrIndexSplitterTest, TestRecoveryHdfs, ChaosMonkeyNothingIsSafeTest, ConvertedLegacyTest, RecoveryZkTest, SuggesterWFSTTest, TestFiltering, TestNoOpRegenerator, TestManagedSchemaFieldResource, TestFieldCollectionResource, SOLR749Test, TestLRUCache, TestLMJelinekMercerSimilarityFactory, QueryElevationComponentTest, URLClassifyProcessorTest, SliceStateUpdateTest, TestDynamicFieldResource, TestCollapseQParserPlugin, SolrCoreCheckLockOnStartupTest, BadComponentTest, SortByFunctionTest, LukeRequestHandlerTest, UpdateRequestProcessorFactoryTest, ShardRoutingCustomTest, TestDefaultSimilarityFactory, AutoCommitTest, UUIDFieldTest, DistributedTermsComponentTest, SpellPossibilityIteratorTest, SolrXmlInZkTest, TestSolrIndexConfig, ResourceLoaderTest, BasicZkTest, TestInfoStreamLogging, SuggesterTest, SolrPluginUtilsTest, TestFaceting, CircularListTest, TestOmitPositions, ChaosMonkeySafeLeaderTest, TestLazyCores, CachingDirectoryFactoryTest, TestClassNameShortening, DirectSolrSpellCheckerTest, HdfsBasicDistributedZk2Test, PreAnalyzedFieldTest, SolrRequestParserTest, LeaderElectionIntegrationTest, DateMathParserTest, TestRandomFaceting, TestRangeQuery, TestHashPartitioner, TestFastOutputStream, StressHdfsTest, TestCoreContainer, DefaultValueUpdateProcessorTest, CoreContainerCoreInitFailuresTest, TestCharFilters, ZkControllerTest, DistributedSuggestComponentTest, TestSuggestSpellingConverter, TestSolrDeletionPolicy1, CursorMarkTest, TestImplicitCoreProperties, DisMaxRequestHandlerTest, TestDistribDocBasedVersion, TestZkChroot, LeaderElectionTest, BinaryUpdateRequestHandlerTest, ParsingFieldUpdateProcessorsTest, DirectSolrConnectionTest, TestFreeTextSuggestions, FullSolrCloudDistribCmdsTest, DocValuesMultiTest, QueryParsingTest, TestUpdate, TestCopyFieldCollectionResource, BasicDistributedZk2Test, TestFoldingMultitermQuery, CoreAdminHandlerTest, UniqFieldsUpdateProcessorFactoryTest, SuggesterFSTTest, TestSurroundQueryParser, MoreLikeThisHandlerTest, TestSerializedLuceneMatchVersion, BasicDistributedZkTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OverseerTest, ClusterStateUpdateTest, ZkSolrClientTest, ZkCLITest, TestDistributedGrouping, TestRecovery, TermVectorComponentDistributedTest, TestRealTimeGet, TestStressReorder, TestJoin, TestMultiCoreConfBootstrap, TestReload, TestStressVersions, HardAutoCommitTest, SimpleFacetsTest, TestSolr4Spatial, TestGroupingSearch, PeerSyncTest, BadIndexSchemaTest, TestFunctionQuery, BasicFunctionalityTest, DirectUpdateHandlerTest, TestBadConfig, OverseerCollectionProcessorTest, HighlighterTest, SoftAutoCommitTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SpellCheckCollatorTest, DocValuesTest, SuggesterTSTTest, TestStressLucene, TestTrie, SpatialFilterTest, TestCSVLoader, PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, DistanceFunctionTest, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, DebugComponentTest, TestQueryUtils, TestWriterPerf, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, TestValueSourceCache, TermVectorComponentTest, TestIndexingPerformance, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, RegexBoostProcessorTest, TestJmxIntegration, TestCollationField, ReturnFieldsTest, TestCSVResponseWriter, MBeansHandlerTest, TestAnalyzedSuggestions, JsonLoaderTest, TestPHPSerializedResponseWriter, PingRequestHandlerTest, SearchHandlerTest, TestLFUCache, TestQuerySenderListener, UpdateParamsTest, AlternateDirectoryTest, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, SolrIndexConfigTest, TestStressRecovery, TestMergePolicyConfig, MultiTermTest, TestDocSet, MinimalSchemaTest, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestCodecSupport, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, TimeZoneUtilsTest, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, PluginInfoTest, TestFastLRUCache, TestSystemIdResolver, PrimUtilsTest, DateFieldTest, SpellingQueryConverterTest, DOMUtilTest, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, TestRTGBase, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestCollationKeyRangeQueries, AliasIntegrationTest, ConnectionManagerTest, DeleteReplicaTest, DeleteShardTest, OverseerRolesTest]
   [junit4] Completed in 34.94s, 1 test, 1 failure <<< FAILURES!

[...truncated 112 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 354 suites, 1541 tests, 1 failure, 39 ignored (5 assumptions)

Total time: 84 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.6.0) - Build # 1204 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1204/
Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 16273 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140117_063000_798.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] 2014-01-17 06:30:36.678 java[4191:af0f] Unable to load realm info from SCDynamicStore
   [junit4] <<< JVM J0: EOF ----

[...truncated 13723 lines...]
-check-forbidden-base:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.6
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.6
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.1
[forbidden-apis] Reading API signatures: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.lang.String#toLowerCase() [Uses default locale]
[forbidden-apis]   in org.apache.solr.cloud.OverseerRolesTest (OverseerRolesTest.java:160)
[forbidden-apis] Scanned 1906 (and 1311 related) class file(s) for forbidden API invocations (in 8.49s), 1 error(s).

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:64: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:271: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:474: Check for forbidden API calls failed, see log.

Total time: 111 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure