You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/05/21 23:47:40 UTC

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 491 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/491/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch

Error Message:
Server at http://127.0.0.1:51327 returned non ok status:500, message:Server Error

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:51327 returned non ok status:500, message:Server Error
	at __randomizedtesting.SeedInfo.seed([32EEEE9975F380AA:B308608102ACE096]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
	at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
	at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
	at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)


REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 230 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 230 seconds
	at __randomizedtesting.SeedInfo.seed([32EEEE9975F380AA:B308608102ACE096]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:512)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:146)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
	at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 9627 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4:junit4]   2> 2394823 T4962 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_/i
[junit4:junit4]   2> 2394831 T4962 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1369170944046
[junit4:junit4]   2> 2394833 T4962 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 2394835 T4963 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 2394936 T4962 oasc.ZkTestServer.run start zk server on port:50624
[junit4:junit4]   2> 2394938 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2394950 T4969 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@337c9816 name:ZooKeeperConnection Watcher:127.0.0.1:50624 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2394950 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2394951 T4962 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 2394963 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2394969 T4971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@719b55f6 name:ZooKeeperConnection Watcher:127.0.0.1:50624/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2394969 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2394970 T4962 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 2394979 T4962 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 2394986 T4962 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 2395000 T4962 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 2395012 T4962 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2395013 T4962 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2395024 T4962 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 2395026 T4962 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 2395034 T4962 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 2395035 T4962 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 2395043 T4962 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 2395044 T4962 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 2395053 T4962 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 2395053 T4962 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 2395069 T4962 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 2395071 T4962 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 2395078 T4962 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 2395080 T4962 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 2395087 T4962 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 2395089 T4962 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 2395097 T4962 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 2395098 T4962 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 2395677 T4962 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2395688 T4962 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50627
[junit4:junit4]   2> 2395688 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2395689 T4962 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2395690 T4962 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323
[junit4:junit4]   2> 2395690 T4962 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/solr.xml
[junit4:junit4]   2> 2395691 T4962 oasc.CoreContainer.<init> New CoreContainer 336967360
[junit4:junit4]   2> 2395692 T4962 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/'
[junit4:junit4]   2> 2395692 T4962 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/'
[junit4:junit4]   2> 2395875 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323
[junit4:junit4]   2> 2395877 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/collection1
[junit4:junit4]   2> 2395879 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/collection1/conf
[junit4:junit4]   2> 2396067 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/collection1/conf/xslt
[junit4:junit4]   2> 2396071 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/collection1/lib
[junit4:junit4]   2> 2396074 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/collection1/lib/classes
[junit4:junit4]   2> 2396077 T4962 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/conf/core.properties, adding to cores
[junit4:junit4]   2> 2396094 T4962 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2396095 T4962 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2396096 T4962 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2396096 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2396097 T4962 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2396097 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2396098 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2396099 T4962 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2396099 T4962 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2396100 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2396131 T4962 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2396132 T4962 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50624/solr
[junit4:junit4]   2> 2396133 T4962 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2396136 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2396141 T4982 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2356db89 name:ZooKeeperConnection Watcher:127.0.0.1:50624 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2396142 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2396147 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2396177 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2396183 T4984 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71250f4 name:ZooKeeperConnection Watcher:127.0.0.1:50624/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2396183 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2396188 T4962 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 2396205 T4962 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 2396214 T4962 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 2396219 T4962 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50627__%2Fi
[junit4:junit4]   2> 2396222 T4962 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50627__%2Fi
[junit4:junit4]   2> 2396231 T4962 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 2396245 T4962 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 2396254 T4962 oasc.Overseer.start Overseer (id=89729986989654019-127.0.0.1:50627__%2Fi-n_0000000000) starting
[junit4:junit4]   2> 2396263 T4962 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 2396280 T4986 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 2396282 T4962 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 2396290 T4962 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 2396295 T4962 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2396304 T4985 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 2396311 T4962 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2396312 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2396312 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2396333 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 2396337 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2396343 T4989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59cba2cd name:ZooKeeperConnection Watcher:127.0.0.1:50624/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2396343 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2396347 T4962 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2396354 T4962 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 2396890 T4962 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2396897 T4962 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50631
[junit4:junit4]   2> 2396900 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2396900 T4962 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2396901 T4962 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568
[junit4:junit4]   2> 2396901 T4962 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/solr.xml
[junit4:junit4]   2> 2396902 T4962 oasc.CoreContainer.<init> New CoreContainer 1750815999
[junit4:junit4]   2> 2396903 T4962 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/'
[junit4:junit4]   2> 2396903 T4962 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/'
[junit4:junit4]   2> 2397074 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568
[junit4:junit4]   2> 2397075 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/collection1
[junit4:junit4]   2> 2397077 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/collection1/conf
[junit4:junit4]   2> 2397255 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/collection1/conf/xslt
[junit4:junit4]   2> 2397262 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/collection1/lib
[junit4:junit4]   2> 2397264 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/collection1/lib/classes
[junit4:junit4]   2> 2397267 T4962 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/conf/core.properties, adding to cores
[junit4:junit4]   2> 2397283 T4962 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2397283 T4962 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2397284 T4962 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2397285 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2397286 T4962 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2397287 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2397287 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2397288 T4962 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2397288 T4962 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2397290 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2397323 T4962 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2397324 T4962 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50624/solr
[junit4:junit4]   2> 2397325 T4962 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2397327 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2397332 T5000 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f00978 name:ZooKeeperConnection Watcher:127.0.0.1:50624 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2397334 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2397341 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2397368 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2397372 T5002 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ef8884b name:ZooKeeperConnection Watcher:127.0.0.1:50624/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2397373 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2397384 T4962 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2398400 T4962 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50631__%2Fi
[junit4:junit4]   2> 2398402 T4962 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50631__%2Fi
[junit4:junit4]   2> 2398417 T4984 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2398418 T5002 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2398419 T4989 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2398430 T4962 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2398431 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2398432 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2398994 T4962 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2399005 T4962 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50634
[junit4:junit4]   2> 2399006 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2399007 T4962 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2399007 T4962 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664
[junit4:junit4]   2> 2399008 T4962 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/solr.xml
[junit4:junit4]   2> 2399008 T4962 oasc.CoreContainer.<init> New CoreContainer 1703647247
[junit4:junit4]   2> 2399009 T4962 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/'
[junit4:junit4]   2> 2399011 T4962 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/'
[junit4:junit4]   2> 2399191 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664
[junit4:junit4]   2> 2399193 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/collection1
[junit4:junit4]   2> 2399195 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/collection1/conf
[junit4:junit4]   2> 2399386 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/collection1/conf/xslt
[junit4:junit4]   2> 2399391 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/collection1/lib
[junit4:junit4]   2> 2399394 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/collection1/lib/classes
[junit4:junit4]   2> 2399397 T4962 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/conf/core.properties, adding to cores
[junit4:junit4]   2> 2399419 T4962 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2399420 T4962 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2399421 T4962 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2399421 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2399422 T4962 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2399423 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2399424 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2399425 T4962 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2399426 T4962 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2399426 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2399454 T4962 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2399455 T4962 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50624/solr
[junit4:junit4]   2> 2399456 T4962 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2399459 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2399465 T5014 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@602d5d1f name:ZooKeeperConnection Watcher:127.0.0.1:50624 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2399466 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2399472 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2399513 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2399517 T5016 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43899af7 name:ZooKeeperConnection Watcher:127.0.0.1:50624/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2399518 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2399538 T4962 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2400546 T4962 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50634__%2Fi
[junit4:junit4]   2> 2400549 T4962 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50634__%2Fi
[junit4:junit4]   2> 2400567 T5016 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2400567 T4984 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2400572 T4989 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2400572 T5002 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2400573 T4962 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2400574 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2400574 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2401118 T4962 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2401130 T4962 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50637
[junit4:junit4]   2> 2401132 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2401133 T4962 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2401134 T4962 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813
[junit4:junit4]   2> 2401134 T4962 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/solr.xml
[junit4:junit4]   2> 2401136 T4962 oasc.CoreContainer.<init> New CoreContainer 534980431
[junit4:junit4]   2> 2401136 T4962 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/'
[junit4:junit4]   2> 2401137 T4962 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/'
[junit4:junit4]   2> 2401318 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813
[junit4:junit4]   2> 2401323 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/collection1
[junit4:junit4]   2> 2401328 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/collection1/conf
[junit4:junit4]   2> 2401505 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/collection1/conf/xslt
[junit4:junit4]   2> 2401510 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/collection1/lib
[junit4:junit4]   2> 2401511 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/collection1/lib/classes
[junit4:junit4]   2> 2401514 T4962 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/conf/core.properties, adding to cores
[junit4:junit4]   2> 2401535 T4962 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2401536 T4962 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2401537 T4962 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2401537 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2401538 T4962 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2401538 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2401540 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2401541 T4962 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2401541 T4962 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2401542 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2401569 T4962 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2401570 T4962 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50624/solr
[junit4:junit4]   2> 2401570 T4962 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2401575 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2401578 T5028 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bdaf54f name:ZooKeeperConnection Watcher:127.0.0.1:50624 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2401579 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2401584 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2401615 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2401618 T5030 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b9c1762 name:ZooKeeperConnection Watcher:127.0.0.1:50624/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2401619 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2401630 T4962 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2402641 T4962 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50637__%2Fi
[junit4:junit4]   2> 2402647 T4962 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50637__%2Fi
[junit4:junit4]   2> 2402664 T5030 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2402666 T5016 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2402667 T5002 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2402667 T4989 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2402678 T4984 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2402678 T4962 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2402679 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2402680 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2403281 T4962 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2403291 T4962 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50640
[junit4:junit4]   2> 2403292 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2403292 T4962 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2403293 T4962 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920
[junit4:junit4]   2> 2403294 T4962 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/solr.xml
[junit4:junit4]   2> 2403294 T4962 oasc.CoreContainer.<init> New CoreContainer 1996972346
[junit4:junit4]   2> 2403295 T4962 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/'
[junit4:junit4]   2> 2403295 T4962 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/'
[junit4:junit4]   2> 2403478 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920
[junit4:junit4]   2> 2403481 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/collection1
[junit4:junit4]   2> 2403482 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/collection1/conf
[junit4:junit4]   2> 2403665 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/collection1/conf/xslt
[junit4:junit4]   2> 2403670 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/collection1/lib
[junit4:junit4]   2> 2403672 T4962 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/collection1/lib/classes
[junit4:junit4]   2> 2403675 T4962 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/conf/core.properties, adding to cores
[junit4:junit4]   2> 2403694 T4962 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2403695 T4962 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2403696 T4962 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2403696 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2403697 T4962 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2403697 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2403698 T4962 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2403699 T4962 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2403699 T4962 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2403700 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2403732 T4962 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2403734 T4962 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50624/solr
[junit4:junit4]   2> 2403734 T4962 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2403740 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2403745 T5042 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@571f132f name:ZooKeeperConnection Watcher:127.0.0.1:50624 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2403747 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2403752 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2403779 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2403789 T5044 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55af34f8 name:ZooKeeperConnection Watcher:127.0.0.1:50624/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2403790 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2403804 T4962 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2404816 T4962 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50640__%2Fi
[junit4:junit4]   2> 2404819 T4962 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50640__%2Fi
[junit4:junit4]   2> 2404830 T5030 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2404832 T5044 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2404848 T5016 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2404849 T4984 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2404849 T5002 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2404850 T4962 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2404850 T4962 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2404851 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2404852 T4989 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2404882 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2404911 T4993 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2
[junit4:junit4]   2> 2404915 T4984 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 2404919 T4984 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 2404921 T4986 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
[junit4:junit4]   2> 	  "operation":"createcollection",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "maxShardsPerNode":null,
[junit4:junit4]   2> 	  "createNodeSet":null,
[junit4:junit4]   2> 	  "name":"nodes_used_collection",
[junit4:junit4]   2> 	  "replicationFactor":"2"}
[junit4:junit4]   2> 2404921 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:50640__%2Fi
[junit4:junit4]   2> 2404923 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:50637__%2Fi
[junit4:junit4]   2> 2404926 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:50634__%2Fi
[junit4:junit4]   2> 2404935 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:50627__%2Fi
[junit4:junit4]   2> 2404943 T5039 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 2404945 T5039 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 2404947 T5024 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2404947 T5024 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 2404950 T5039 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 2404953 T5039 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2404951 T5009 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 2404954 T5009 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 2404955 T4980 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2404955 T4980 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 2404953 T5024 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 2404957 T5024 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2404959 T5009 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 2404960 T5009 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2404960 T4980 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 2404962 T4980 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2404963 T5039 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2404964 T5039 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 2404965 T5024 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2404966 T5024 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 2404973 T5009 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2404974 T5009 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 2404975 T4980 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2404976 T4980 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 2404988 T5024 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 2404991 T5039 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 2404994 T5009 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 2405000 T4980 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 2405001 T5024 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/'
[junit4:junit4]   2> 2405013 T5039 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/nodes_used_collection_shard1_replica1/'
[junit4:junit4]   2> 2405017 T5009 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/nodes_used_collection_shard2_replica1/'
[junit4:junit4]   2> 2405024 T4980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/'
[junit4:junit4]   2> 2405362 T5039 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2405364 T4980 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2405367 T5024 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2405385 T5009 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2405635 T5009 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2405650 T5009 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2405668 T5009 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
[junit4:junit4]   2> 2405687 T4980 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2405687 T5039 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2405690 T5024 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2405695 T4980 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2405702 T5039 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2405706 T4980 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
[junit4:junit4]   2> 2405718 T5039 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
[junit4:junit4]   2> 2405726 T5024 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2405735 T5024 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
[junit4:junit4]   2> 2408768 T5009 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2408800 T5039 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2408805 T5009 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2408814 T5009 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2408823 T5039 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2408824 T5024 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2408835 T5039 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2408856 T4980 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2408866 T5024 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2408891 T4980 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2408900 T4980 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2408922 T5009 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2408922 T5024 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2408941 T5009 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2408946 T5039 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2408962 T5009 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2408969 T5009 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2408970 T5009 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2408971 T5009 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2408982 T5009 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2408983 T5009 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2408983 T5009 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2408982 T5039 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2408984 T5009 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/nodes_used_collection_shard2_replica1/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/nodes_used_collection_shard2_replica1/data/
[junit4:junit4]   2> 2408986 T5009 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6d56bbcf
[junit4:junit4]   2> 2408987 T5009 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2408989 T5009 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/nodes_used_collection_shard2_replica1/data
[junit4:junit4]   2> 2408990 T5009 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/nodes_used_collection_shard2_replica1/data/index/
[junit4:junit4]   2> 2408991 T5009 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2408996 T5009 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/nodes_used_collection_shard2_replica1/data/index
[junit4:junit4]   2> 2409000 T5039 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2409005 T5009 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@33a8092e lockFactory=org.apache.lucene.store.NativeFSLockFactory@2684d6df),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2409006 T5009 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2409008 T5039 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2409014 T5039 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2409015 T5039 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2409016 T5009 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2409017 T5009 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2409018 T5009 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2409019 T5009 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2409020 T5009 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2409021 T5009 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2409025 T5009 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2409026 T5009 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2409031 T5009 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2409029 T5039 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2409038 T5039 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2409038 T5039 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2409039 T5039 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/nodes_used_collection_shard1_replica1/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/nodes_used_collection_shard1_replica1/data/
[junit4:junit4]   2> 2409039 T5039 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6d56bbcf
[junit4:junit4]   2> 2409040 T5039 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2409041 T5039 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/nodes_used_collection_shard1_replica1/data
[junit4:junit4]   2> 2409043 T5039 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/nodes_used_collection_shard1_replica1/data/index/
[junit4:junit4]   2> 2409043 T5039 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2409044 T5039 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/nodes_used_collection_shard1_replica1/data/index
[junit4:junit4]   2> 2409053 T5039 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@222b1722 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d30ccea),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2409054 T5039 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2409059 T5039 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2409065 T5039 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2409066 T5039 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2409067 T5039 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2409069 T5039 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2409069 T5039 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2409070 T5039 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2409071 T5039 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2409071 T5039 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2409074 T5009 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2409065 T4980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2409090 T5009 oass.SolrIndexSearcher.<init> Opening Searcher@3d66d1f2 main
[junit4:junit4]   2> 2409104 T5039 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2409111 T5009 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/nodes_used_collection_shard2_replica1/data/tlog
[junit4:junit4]   2> 2409114 T5009 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2409115 T5009 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2409116 T5039 oass.SolrIndexSearcher.<init> Opening Searcher@7839e0c1 main
[junit4:junit4]   2> 2409117 T5039 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/nodes_used_collection_shard1_replica1/data/tlog
[junit4:junit4]   2> 2409119 T5039 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2409120 T5039 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2409125 T5050 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher@3d66d1f2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2409127 T5024 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2409135 T4980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2409140 T5051 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher@7839e0c1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2409148 T5039 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down
[junit4:junit4]   2> 2409152 T5009 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down
[junit4:junit4]   2> 2409164 T4980 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2409165 T5024 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2409169 T4980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2409170 T4980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2409171 T4980 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2409176 T5024 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2409182 T4980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2409182 T4980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2409183 T4980 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2409184 T4980 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/
[junit4:junit4]   2> 2409184 T4980 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6d56bbcf
[junit4:junit4]   2> 2409186 T4980 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2409187 T4980 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data
[junit4:junit4]   2> 2409182 T5024 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2409188 T5024 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2409189 T5024 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2409190 T4980 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/index/
[junit4:junit4]   2> 2409191 T4980 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2409193 T4980 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/index
[junit4:junit4]   2> 2409198 T5024 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2409199 T5024 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2409200 T5024 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2409200 T4980 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@74a44684 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7f1f0b26),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2409201 T4980 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2409200 T5024 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/
[junit4:junit4]   2> 2409204 T5024 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6d56bbcf
[junit4:junit4]   2> 2409205 T5024 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2409209 T5024 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data
[junit4:junit4]   2> 2409211 T5024 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/index/
[junit4:junit4]   2> 2409214 T5024 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2409215 T5024 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/index
[junit4:junit4]   2> 2409218 T4980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2409219 T4980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2409223 T4980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2409225 T4980 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2409226 T5024 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7dacb2d7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6632e8fd),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2409226 T5024 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2409226 T4980 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2409228 T4980 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2409228 T4980 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2409229 T4980 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2409230 T4980 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2409231 T5024 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2409232 T5024 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2409233 T5024 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2409235 T5024 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2409236 T5024 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2409236 T5024 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2409237 T5024 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2409239 T5024 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2409240 T5024 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2409254 T5024 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2409254 T4980 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2409266 T5024 oass.SolrIndexSearcher.<init> Opening Searcher@11c5f074 main
[junit4:junit4]   2> 2409267 T5024 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/tlog
[junit4:junit4]   2> 2409269 T4980 oass.SolrIndexSearcher.<init> Opening Searcher@3e9b793 main
[junit4:junit4]   2> 2409270 T4980 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/tlog
[junit4:junit4]   2> 2409272 T4980 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2409273 T4980 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2409270 T5024 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2409275 T5024 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2409280 T5053 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher@3e9b793 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2409284 T5052 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher@11c5f074 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2409337 T5024 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down
[junit4:junit4]   2> 2409338 T4980 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down
[junit4:junit4]   2> 2409927 T4985 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2409928 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50640__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50640/_/i"}
[junit4:junit4]   2> 2409929 T4985 oasc.Overseer$ClusterStateUpdater.createCollection Create collection nodes_used_collection with numShards 2
[junit4:junit4]   2> 2409939 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50634__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50634/_/i"}
[junit4:junit4]   2> 2409948 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50637__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50637/_/i"}
[junit4:junit4]   2> 2409959 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50627__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50627/_/i"}
[junit4:junit4]   2> 2409980 T5044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2409980 T4989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2409981 T5030 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2409981 T5016 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2409982 T5002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2409982 T4984 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2410160 T5039 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 2410160 T5039 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica1 address:http://127.0.0.1:50640/_/i collection:nodes_used_collection shard:shard1
[junit4:junit4]   2> 2410163 T5039 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2410164 T5009 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 2410165 T5009 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica1 address:http://127.0.0.1:50634/_/i collection:nodes_used_collection shard:shard2
[junit4:junit4]   2> 2410167 T5009 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
[junit4:junit4]   2> 2410194 T5039 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2410196 T5009 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2410208 T5039 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=2 found=1 timeoutin=179997
[junit4:junit4]   2> 2410209 T5009 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=2 found=1 timeoutin=179998
[junit4:junit4]   2> 2410343 T5024 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2410344 T5024 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica2 address:http://127.0.0.1:50637/_/i collection:nodes_used_collection shard:shard1
[junit4:junit4]   2> 2410343 T4980 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2410345 T4980 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica2 address:http://127.0.0.1:50627/_/i collection:nodes_used_collection shard:shard2
[junit4:junit4]   2> 2410713 T5039 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2410713 T5039 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2410714 T5039 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50640/_/i/nodes_used_collection_shard1_replica1/
[junit4:junit4]   2> 2410714 T5039 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:50640/_/i START replicas=[http://127.0.0.1:50637/_/i/nodes_used_collection_shard1_replica2/] nUpdates=100
[junit4:junit4]   2> 2410715 T5009 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2410716 T5009 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2410716 T5009 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50634/_/i/nodes_used_collection_shard2_replica1/
[junit4:junit4]   2> 2410717 T5009 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica1 url=http://127.0.0.1:50634/_/i START replicas=[http://127.0.0.1:50627/_/i/nodes_used_collection_shard2_replica2/] nUpdates=100
[junit4:junit4]   2> 2410718 T5039 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:50640/_/i DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 2410718 T5039 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4]   2> 2410719 T5039 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4]   2> 2410719 T5039 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50640/_/i/nodes_used_collection_shard1_replica1/
[junit4:junit4]   2> 2410720 T5039 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard1
[junit4:junit4]   2> 2410721 T5009 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica1 url=http://127.0.0.1:50634/_/i DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 2410722 T5009 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4]   2> 2410722 T5009 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4]   2> 2410723 T5009 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50634/_/i/nodes_used_collection_shard2_replica1/
[junit4:junit4]   2> 2410723 T5009 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard2
[junit4:junit4]   2> ASYNC  NEW_CORE C1031 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore@4d1c6a7b url=http://127.0.0.1:50627/_/i/nodes_used_collection_shard2_replica2 node=127.0.0.1:50627__%2Fi C1031_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=down, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:50627__%2Fi, base_url=http://127.0.0.1:50627/_/i}
[junit4:junit4]   2> 2410751 T4979 C1031 P50627 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/_/i path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=5 
[junit4:junit4]   2> ASYNC  NEW_CORE C1032 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore@7c97b44 url=http://127.0.0.1:50637/_/i/nodes_used_collection_shard1_replica2 node=127.0.0.1:50637__%2Fi C1032_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=down, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:50637__%2Fi, base_url=http://127.0.0.1:50637/_/i}
[junit4:junit4]   2> 2410770 T5023 C1032 P50637 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/_/i path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 2411528 T4985 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2411599 T5044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2411599 T4984 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2411600 T4989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2411600 T5002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2411600 T5016 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2411599 T5030 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2411612 T5009 oasc.ZkController.register We are http://127.0.0.1:50634/_/i/nodes_used_collection_shard2_replica1/ and leader is http://127.0.0.1:50634/_/i/nodes_used_collection_shard2_replica1/
[junit4:junit4]   2> 2411613 T5009 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica1 baseURL=http://127.0.0.1:50634/_/i
[junit4:junit4]   2> 2411614 T5009 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2411614 T5009 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=active
[junit4:junit4]   2> 2411613 T5039 oasc.ZkController.register We are http://127.0.0.1:50640/_/i/nodes_used_collection_shard1_replica1/ and leader is http://127.0.0.1:50640/_/i/nodes_used_collection_shard1_replica1/
[junit4:junit4]   2> 2411615 T5039 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica1 baseURL=http://127.0.0.1:50640/_/i
[junit4:junit4]   2> 2411616 T5039 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2411616 T5039 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=active
[junit4:junit4]   2> 2411616 T5024 oasc.ZkController.register We are http://127.0.0.1:50637/_/i/nodes_used_collection_shard1_replica2/ and leader is http://127.0.0.1:50640/_/i/nodes_used_collection_shard1_replica1/
[junit4:junit4]   2> 2411618 T5024 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica2 baseURL=http://127.0.0.1:50637/_/i
[junit4:junit4]   2> 2411618 T5024 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2411619 T5024 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> 2411620 T5056 C1032 P50637 oasc.RecoveryStrategy.run Starting recovery process.  core=nodes_used_collection_shard1_replica2 recoveringAfterStartup=true
[junit4:junit4]   2> 2411621 T5056 C1032 P50637 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 2411622 T5056 C1032 P50637 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=recovering
[junit4:junit4]   2> 2411626 T5024 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2411628 T5009 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2411629 T5056 C1032 P50637 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2411630 T5024 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=6683 
[junit4:junit4]   2> 2411631 T5039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2411645 T5009 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=6694 
[junit4:junit4]   2> 2411645 T5039 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=6703 
[junit4:junit4]   2> 2411664 T4980 oasc.ZkController.register We are http://127.0.0.1:50627/_/i/nodes_used_collection_shard2_replica2/ and leader is http://127.0.0.1:50634/_/i/nodes_used_collection_shard2_replica1/
[junit4:junit4]   2> 2411665 T4980 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica2 baseURL=http://127.0.0.1:50627/_/i
[junit4:junit4]   2> 2411666 T4980 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2411666 T4980 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> 2411668 T5057 C1031 P50627 oasc.RecoveryStrategy.run Starting recovery process.  core=nodes_used_collection_shard2_replica2 recoveringAfterStartup=true
[junit4:junit4]   2> 2411668 T5057 C1031 P50627 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 2411670 T5057 C1031 P50627 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=recovering
[junit4:junit4]   2> 2411671 T4980 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2411674 T5057 C1031 P50627 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2411675 T4980 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=6720 
[junit4:junit4]   2> 2411679 T4986 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: nodes_used_collection
[junit4:junit4]   2> 2411699 T5002 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
[junit4:junit4]   2> 2411709 T4986 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=6683},core=nodes_used_collection_shard1_replica2},null={responseHeader={status=0,QTime=6694},core=nodes_used_collection_shard2_replica1},null={responseHeader={status=0,QTime=6703},core=nodes_used_collection_shard1_replica1},null={responseHeader={status=0,QTime=6720},core=nodes_used_collection_shard2_replica2}}}
[junit4:junit4]   2> 2411709 T4993 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2} status=0 QTime=6798 
[junit4:junit4]   2> 2411713 T5040 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:50637__%2Fi_nodes_used_collection_shard1_replica2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 2411713 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 2411716 T5010 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:50627__%2Fi_nodes_used_collection_shard2_replica2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 2411719 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2411731 T5059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14e7556 name:ZooKeeperConnection Watcher:127.0.0.1:50624/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2411734 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2411738 T4962 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2411746 T4962 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 2411750 T4962 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2411756 T5061 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72532496 name:ZooKeeperConnection Watcher:127.0.0.1:50624/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2411756 T4962 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2411760 T4962 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2411791 T5036 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=5&maxShardsPerNode=6&name=awholynewcollection_0&replicationFactor=5&action=CREATE&wt=javabin&version=2
[junit4:junit4]   2> 2411798 T4984 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 2411805 T4984 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 2411806 T4986 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
[junit4:junit4]   2> 	  "operation":"createcollection",
[junit4:junit4]   2> 	  "numShards":"5",
[junit4:junit4]   2> 	  "maxShardsPerNode":"6",
[junit4:junit4]   2> 	  "createNodeSet":null,
[junit4:junit4]   2> 	  "name":"awholynewcollection_0",
[junit4:junit4]   2> 	  "replicationFactor":"5"}
[junit4:junit4]   2> 2411807 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica1 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:50640__%2Fi
[junit4:junit4]   2> 2411807 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica2 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:50634__%2Fi
[junit4:junit4]   2> 2411808 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica3 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:50631__%2Fi
[junit4:junit4]   2> 2411808 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica4 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:50637__%2Fi
[junit4:junit4]   2> 2411808 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica5 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:50627__%2Fi
[junit4:junit4]   2> 2411813 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica1 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:50640__%2Fi
[junit4:junit4]   2> 2411814 T5011 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard1_replica2
[junit4:junit4]   2> 2411815 T5011 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411816 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica2 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:50634__%2Fi
[junit4:junit4]   2> 2411817 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica3 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:50631__%2Fi
[junit4:junit4]   2> 2411813 T5037 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard1_replica1
[junit4:junit4]   2> 2411827 T5037 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411831 T4997 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica3' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard2_replica3
[junit4:junit4]   2> 2411831 T4997 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411824 T4977 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica5' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard1_replica5
[junit4:junit4]   2> 2411833 T4977 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411823 T4995 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica3' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard1_replica3
[junit4:junit4]   2> 2411834 T4995 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411820 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica4 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:50637__%2Fi
[junit4:junit4]   2> 2411835 T5037 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411835 T5037 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411816 T5021 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica4' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard1_replica4
[junit4:junit4]   2> 2411837 T5021 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411837 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica5 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:50627__%2Fi
[junit4:junit4]   2> 2411831 T5039 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard2_replica1
[junit4:junit4]   2> 2411839 T5039 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411828 T5007 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard2_replica2
[junit4:junit4]   2> 2411843 T5007 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411828 T5011 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411847 T5011 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411847 T4995 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411849 T4995 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411847 T4997 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411850 T4997 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411851 T5022 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica4' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard2_replica4
[junit4:junit4]   2> 2411851 T5022 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411847 T4978 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica5' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard2_replica5
[junit4:junit4]   2> 2411853 T4978 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411840 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica1 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:50640__%2Fi
[junit4:junit4]   2> 2411854 T5039 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411855 T5039 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411855 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica2 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:50634__%2Fi
[junit4:junit4]   2> 2411839 T4977 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411858 T4977 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411852 T5037 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2411851 T5021 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411866 T5021 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411867 T5008 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard3_replica2
[junit4:junit4]   2> 2411868 T5008 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411848 T5007 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411868 T4997 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2411870 T4997 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411867 T4995 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2411871 T4995 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411866 T5037 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411865 T4978 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411875 T4978 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411863 T5038 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard3_replica1
[junit4:junit4]   2> 2411876 T5038 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411863 T5011 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2411877 T5011 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411878 T5021 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2411878 T5021 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411859 T5022 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411858 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica3 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:50631__%2Fi
[junit4:junit4]   2> 2411880 T5022 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411876 T4977 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2411884 T4977 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411885 T5038 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411886 T5038 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411874 T5008 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 2411888 T5008 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411889 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica4 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:50637__%2Fi
[junit4:junit4]   2> 2411870 T5039 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2411892 T5039 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411893 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica5 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:50627__%2Fi
[junit4:junit4]   2> 2411902 T4995 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411902 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica1 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:50640__%2Fi
[junit4:junit4]   2> 2411869 T5007 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 2411902 T5023 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica4' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard3_replica4
[junit4:junit4]   2> 2411906 T5023 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411907 T5038 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411908 T4997 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411898 T4998 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica3' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard3_replica3
[junit4:junit4]   2> 2411916 T4998 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411893 T4978 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 2411918 T5008 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411917 T5011 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411916 T5037 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411916 T4977 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411915 T5035 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard4_replica1
[junit4:junit4]   2> 2411923 T5035 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411913 T4979 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica5' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard3_replica5
[junit4:junit4]   2> 2411925 T4979 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411907 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica2 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:50634__%2Fi
[junit4:junit4]   2> 2411927 T5021 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411922 T5023 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2411930 T5023 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411918 T5022 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411918 T4978 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411936 T5035 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2411937 T5035 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411935 T5009 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard4_replica2
[junit4:junit4]   2> 2411938 T5009 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411940 T5039 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411942 T4995 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard1_replica3/'
[junit4:junit4]   2> 2411944 T4979 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2411945 T4979 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411933 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica3 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:50631__%2Fi
[junit4:junit4]   2> 2411927 T5007 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 2411941 T4998 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2411947 T4998 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411948 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica4 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:50637__%2Fi
[junit4:junit4]   2> 2411950 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica5 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:50627__%2Fi
[junit4:junit4]   2> 2411956 T4994 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica3' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard4_replica3
[junit4:junit4]   2> 2411968 T4994 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411968 T4976 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica5' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard4_replica5
[junit4:junit4]   2> 2411969 T4976 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411961 T5009 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2411971 T5009 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411961 T5025 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica4' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard4_replica4
[junit4:junit4]   2> 2411973 T5025 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2411974 T4997 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard2_replica3/'
[junit4:junit4]   2> 2411975 T5037 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard1_replica1/'
[junit4:junit4]   2> 2411976 T5011 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard1_replica2/'
[junit4:junit4]   2> 2411977 T5008 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2411984 T5023 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard3_replica4/'
[junit4:junit4]   2> 2411989 T4977 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard1_replica5/'
[junit4:junit4]   2> 2411958 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard5_replica1 as part of slice shard5 of collection awholynewcollection_0 on 127.0.0.1:50640__%2Fi
[junit4:junit4]   2> 2412031 T4976 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2412032 T4976 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412030 T5008 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard3_replica2/'
[junit4:junit4]   2> 2412035 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard5_replica2 as part of slice shard5 of collection awholynewcollection_0 on 127.0.0.1:50634__%2Fi
[junit4:junit4]   2> 2412028 T5039 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard2_replica1/'
[junit4:junit4]   2> 2412027 T5021 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard1_replica4/'
[junit4:junit4]   2> 2412025 T5025 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2412052 T5025 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412024 T4994 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2412053 T4994 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412023 T4998 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard3_replica3/'
[junit4:junit4]   2> 2412022 T5035 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard4_replica1/'
[junit4:junit4]   2> 2411972 T5038 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412079 T5022 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412065 T4979 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard3_replica5/'
[junit4:junit4]   2> 2412063 T5007 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412044 T5012 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard5_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard5_replica2
[junit4:junit4]   2> 2412082 T5012 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2412043 T5080 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard5_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard5_replica1
[junit4:junit4]   2> 2412083 T5080 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2412038 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard5_replica3 as part of slice shard5 of collection awholynewcollection_0 on 127.0.0.1:50631__%2Fi
[junit4:junit4]   2> 2412032 T5009 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard4_replica2/'
[junit4:junit4]   2> 2412087 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard5_replica4 as part of slice shard5 of collection awholynewcollection_0 on 127.0.0.1:50637__%2Fi
[junit4:junit4]   2> 2412090 T4986 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard5_replica5 as part of slice shard5 of collection awholynewcollection_0 on 127.0.0.1:50627__%2Fi
[junit4:junit4]   2> 2412096 T5024 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard5_replica4' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard5_replica4
[junit4:junit4]   2> 2412096 T5024 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2412092 T4993 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard5_replica3' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard5_replica3
[junit4:junit4]   2> 2412102 T4993 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2412092 T4978 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412127 T5024 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2412130 T5024 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412127 T4994 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard4_replica3/'
[junit4:junit4]   2> 2412124 T5025 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard4_replica4/'
[junit4:junit4]   2> 2412123 T5012 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2412133 T5012 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412123 T4993 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2412134 T4993 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412113 T4976 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard4_replica5/'
[junit4:junit4]   2> 2412112 T4980 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard5_replica5' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard5_replica5
[junit4:junit4]   2> 2412143 T4980 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 2412130 T5080 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2412144 T5080 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412148 T5022 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard2_replica4/'
[junit4:junit4]   2> 2412162 T4980 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2412162 T4980 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 2412179 T4978 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard2_replica5/'
[junit4:junit4]   2> 2412182 T5024 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/awholynewcollection_0_shard5_replica4/'
[junit4:junit4]   2> 2412187 T4980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard5_replica5/'
[junit4:junit4]   2> 2412197 T4993 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369170945568/awholynewcollection_0_shard5_replica3/'
[junit4:junit4]   2> 2412204 T5007 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard2_replica2/'
[junit4:junit4]   2> 2412208 T5038 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard3_replica1/'
[junit4:junit4]   2> 2412233 T5080 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369170951920/awholynewcollection_0_shard5_replica1/'
[junit4:junit4]   2> 2412249 T5012 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369170947664/awholynewcollection_0_shard5_replica2/'
[junit4:junit4]   2> 2412506 T5022 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2412582 T4978 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2412601 T5009 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2412744 T4995 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2412768 T5007 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2412843 T5012 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2412882 T5023 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413000 T4994 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413111 T5039 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413144 T4979 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413161 T4985 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2413164 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50634__%2Fi_nodes_used_collection_shard2_replica1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50634__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50634/_/i"}
[junit4:junit4]   2> 2413209 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50640__%2Fi_nodes_used_collection_shard1_replica1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50640__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50640/_/i"}
[junit4:junit4]   2> 2413227 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50637__%2Fi_nodes_used_collection_shard1_replica2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50637__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50637/_/i"}
[junit4:junit4]   2> 2413264 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50627__%2Fi_nodes_used_collection_shard2_replica2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50627__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50627/_/i"}
[junit4:junit4]   2> 2413298 T5059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2413299 T5061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2413300 T4989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2413303 T5044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2413316 T5030 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2413316 T5002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2413318 T5016 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2413321 T4984 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2413343 T4998 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413382 T5009 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2413436 T4980 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413474 T5080 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413480 T5009 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2413524 T5009 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica2] Schema name=test
[junit4:junit4]   2> 2413591 T4977 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413716 T5040 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:50637__%2Fi_nodes_used_collection_shard1_replica2, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4]   2> 2413717 T5040 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:50637__%252Fi_nodes_used_collection_shard1_replica2&state=recovering&nodeName=127.0.0.1:50637__%252Fi&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard1_replica1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2004 
[junit4:junit4]   2> 2413718 T5010 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:50627__%2Fi_nodes_used_collection_shard2_replica2, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4]   2> 2413719 T5010 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:50627__%252Fi_nodes_used_collection_shard2_replica2&state=recovering&nodeName=127.0.0.1:50627__%252Fi&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard2_replica1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2004 
[junit4:junit4]   2> 2413721 T4976 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413836 T5035 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2413875 T5021 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2414030 T4993 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2414051 T5022 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2414065 T5022 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2414083 T5008 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2414197 T5022 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica4] Schema name=test
[junit4:junit4]   2> 2414280 T5038 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2414354 T5007 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2414354 T5024 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2414377 T5007 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2414449 T4995 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2414457 T5025 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2414464 T4978 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2414469 T4978 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2414478 T4995 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2414498 T4978 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica5] Schema name=test
[junit4:junit4]   2> 2414513 T4995 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica3] Schema name=test
[junit4:junit4]   2> 2414555 T5037 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2414589 T5007 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica2] Schema name=test
[junit4:junit4]   2> 2414648 T4994 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2414658 T4994 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2414671 T4994 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica3] Schema name=test
[junit4:junit4]   2> 2414722 T5011 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2414907 T5080 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2414921 T5080 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2414939 T5080 oass.IndexSchema.readSchema [awholynewcollection_0_shard5_replica1] Schema name=test
[junit4:junit4]   2> 2415083 T5023 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2415092 T5023 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2415131 T4997 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2415232 T5023 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica4] Schema name=test
[junit4:junit4]   2> 2415311 T5012 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2415318 T5012 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2415464 T5039 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2415485 T5039 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2415515 T4980 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2415525 T5012 oass.IndexSchema.readSchema [awholynewcollection_0_shard5_replica2] Schema name=test
[junit4:junit4]   2> 2415541 T4980 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2415569 T4980 oass.IndexSchema.readSchema [awholynewcollection_0_shard5_replica5] Schema name=test
[junit4:junit4]   2> ASYNC  NEW_CORE C1033 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore@7c97b44 url=http://127.0.0.1:50637/_/i/nodes_used_collection_shard1_replica2 node=127.0.0.1:50637__%2Fi C1033_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:50637__%2Fi, base_url=http://127.0.0.1:50637/_/i}
[junit4:junit4]   2> 2415720 T5056 C1033 P50637 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50640/_/i/nodes_used_collection_shard1_replica1/ core=nodes_used_collection_shard1_replica2 - recoveringAfterStartup=true
[junit4:junit4]   2> 2415720 T5056 C1033 P50637 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica2 url=http://127.0.0.1:50637/_/i START replicas=[http://127.0.0.1:50640/_/i/nodes_used_collection_shard1_replica1/] nUpdates=100
[junit4:junit4]   2> 2415721 T5056 C1033 P50637 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 2415722 T5056 C1033 P50637 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2415722 T5056 C1033 P50637 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2415722 T5056 C1033 P50637 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2415723 T5056 C1033 P50637 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 2415723 T5056 C1033 P50637 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50640/_/i/nodes_used_collection_shard1_replica1/. core=nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2415724 T5056 C1033 P50637 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2415725 T4998 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> ASYNC  NEW_CORE C1034 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore@4d1c6a7b url=http://127.0.0.1:50627/_/i/nodes_used_collection_shard2_replica2 node=127.0.0.1:50627__%2Fi C1034_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:50627__%2Fi, base_url=http://127.0.0.1:50627/_/i}
[junit4:junit4]   2> 2415728 T5057 C1034 P50627 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50634/_/i/nodes_used_collection_shard2_replica1/ core=nodes_used_collection_shard2_replica2 - recoveringAfterStartup=true
[junit4:junit4]   2> 2415728 T5057 C1034 P50627 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica2 url=http://127.0.0.1:50627/_/i START replicas=[http://127.0.0.1:50634/_/i/nodes_used_collection_shard2_replica1/] nUpdates=100
[junit4:junit4]   2> 2415733 T5057 C1034 P50627 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 2415801 T5057 C1034 P50627 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2415801 T5057 C1034 P50627 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2415801 T5057 C1034 P50627 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2415802 T5057 C1034 P50627 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 2415802 T5057 C1034 P50627 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50634/_/i/nodes_used_collection_shard2_replica1/. core=nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2415802 T5057 C1034 P50627 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2415809 T5038 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2415790 T5008 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2415771 T4977 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2415752 T4979 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> ASYNC  NEW_CORE C1035 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore@4facf511 url=http://127.0.0.1:50634/_/i/nodes_used_collection_shard2_replica1 node=127.0.0.1:50634__%2Fi C1035_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:50634__%2Fi, base_url=http://127.0.0.1:50634/_/i, leader=true}
[junit4:junit4]   2> 2415898 T5087 C1035 P50634 oasc.SolrCore.execute [nodes_used_collection_shard2_replica1] webapp=/_/i path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=11 
[junit4:junit4]   2> 2415813 T5039 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica1] Schema name=test
[junit4:junit4]   2> 2415926 T4998 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2415934 T4979 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2415942 T4998 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica3] Schema name=test
[junit4:junit4]   2> 2415943 T5008 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2415944 T4977 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2415956 T4977 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica5] Schema name=test
[junit4:junit4]   2> 2415968 T5038 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2416029 T4979 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica5] Schema name=test
[junit4:junit4]   2> 2416033 T5021 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2416037 T5021 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2416046 T5021 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica4] Schema name=test
[junit4:junit4]   2> 2416074 T5010 C1035 P50634 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2416093 T5010 C1035 P50634 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@33a8092e lockFactory=org.apache.lucene.store.NativeFSLockFactory@2684d6df),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2416095 T5010 C1035 P50634 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2416109 T5010 C1035 P50634 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@33a8092e lockFactory=org.apache.lucene.store.NativeFSLockFactory@2684d6df),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@33a8092e lockFactory=org.apache.lucene.store.NativeFSLockFactory@2684d6df),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2416110 T5010 C1035 P50634 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2416114 T5010 C1035 P50634 oass.SolrIndexSearcher.<init> Opening Searcher@816d2fa realtime
[junit4:junit4]   2> 2416117 T5010 C1035 P50634 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2416119 T5010 C1035 P50634 oasup.LogUpdateProcessor.finish [nodes_used_collection_shard2_replica1] webapp=/_/i path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 45
[junit4:junit4]   2> 2416121 T5057 C1034 P50627 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2416122 T5057 C1034 P50627 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 2416137 T5010 C1035 P50634 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2416138 T5010 C1035 P50634 oasc.SolrCore.execute [nodes_used_collection_shard2_replica1] webapp=/_/i path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 2416149 T4993 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2416150 T5057 C1034 P50627 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 2416164 T5057 C1034 P50627 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 2416164 T5057 C1034 P50627 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 2416168 T4993 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2416180 T5010 C1035 P50634 oasc.SolrCore.execute [nodes_used_collection_shard2_replica1] webapp=/_/i path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 2416195 T5057 C1034 P50627 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> ASYNC  NEW_CORE C1036 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore@720e9c23 url=http://127.0.0.1:50640/_/i/nodes_used_collection_shard1_replica1 node=127.0.0.1:50640__%2Fi C1036_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=active, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:50640__%2Fi, base_url=http://127.0.0.1:50640/_/i, leader=true}
[junit4:junit4]   2> 2416199 T5086 C1036 P50640 oasc.SolrCore.execute [nodes_used_collection_shard1_replica1] webapp=/_/i path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=4 
[junit4:junit4]   2> 2416224 T5057 C1034 P50627 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/index.20130522001605410
[junit4:junit4]   2> 2416224 T5057 C1034 P50627 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@2fe0f2da lockFactory=org.apache.lucene.store.NativeFSLockFactory@36dbbd30) fullCopy=false
[junit4:junit4]   2> 2416229 T5040 C1036 P50640 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2416232 T5040 C1036 P50640 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@222b1722 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d30ccea),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2416234 T5040 C1036 P50640 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2416236 T5087 C1035 P50634 oasc.SolrCore.execute [nodes_used_collection_shard2_replica1] webapp=/_/i path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1 
[junit4:junit4]   2> 2416238 T5040 C1036 P50640 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@222b1722 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d30ccea),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@222b1722 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d30ccea),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2416238 T5040 C1036 P50640 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2416239 T5040 C1036 P50640 oass.SolrIndexSearcher.<init> Opening Searcher@570b9d37 realtime
[junit4:junit4]   2> 2416240 T5040 C1036 P50640 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2416243 T5040 C1036 P50640 oasup.LogUpdateProcessor.finish [nodes_used_collection_shard1_replica1] webapp=/_/i path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 24
[junit4:junit4]   2> 2416245 T5056 C1033 P50637 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2416245 T5056 C1033 P50637 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 2416252 T5090 C1036 P50640 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2416253 T5090 C1036 P50640 oasc.SolrCore.execute [nodes_used_collection_shard1_replica1] webapp=/_/i path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 2416254 T5057 C1034 P50627 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 2416254 T5056 C1033 P50637 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 2416255 T5056 C1033 P50637 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 2416255 T5056 C1033 P50637 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 2416256 T5038 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica1] Schema name=test
[junit4:junit4]   2> 2416267 T5040 C1036 P50640 oasc.SolrCore.execute [nodes_used_collection_shard1_replica1] webapp=/_/i path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1 
[junit4:junit4]   2> 2416268 T5057 C1034 P50627 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 2416269 T5057 C1034 P50627 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2416271 T5057 C1034 P50627 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@74a44684 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7f1f0b26),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@74a44684 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7f1f0b26),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2416273 T5057 C1034 P50627 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2416273 T5057 C1034 P50627 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 2416274 T5057 C1034 P50627 oass.SolrIndexSearcher.<init> Opening Searcher@1b260dbd main
[junit4:junit4]   2> 2416276 T5056 C1033 P50637 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 2416276 T5053 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher@1b260dbd main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2416281 T5056 C1033 P50637 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/index.20130522001605489
[junit4:junit4]   2> 2416282 T5056 C1033 P50637 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@51f7c19f lockFactory=org.apache.lucene.store.NativeFSLockFactory@37b263e5) fullCopy=false
[junit4:junit4]   2> 2416287 T5057 C1034 P50627 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/index.20130522001605410 [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/index.20130522001605410;done=true>>]
[junit4:junit4]   2> 2416288 T5057 C1034 P50627 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/index.20130522001605410
[junit4:junit4]   2> 2416288 T5057 C1034 P50627 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/nodes_used_collection_shard2_replica2/data/index.20130522001605410
[junit4:junit4]   2> 2416289 T5057 C1034 P50627 oasc.RecoveryStrategy.replay No replay needed. core=nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2416289 T5057 C1034 P50627 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2416289 T5057 C1034 P50627 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=active
[junit4:junit4]   2> 2416290 T5086 C1036 P50640 oasc.SolrCore.execute [nodes_used_collection_shard1_replica1] webapp=/_/i path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 2416303 T4993 oass.IndexSchema.readSchema [awholynewcollection_0_shard5_replica3] Schema name=test
[junit4:junit4]   2> 2416306 T5056 C1033 P50637 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 2416309 T5057 C1034 P50627 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 2416335 T5008 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica2] Schema name=test
[junit4:junit4]   2> 2416375 T5024 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2416381 T5056 C1033 P50637 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 2416384 T5056 C1033 P50637 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2416406 T5024 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2416424 T5024 oass.IndexSchema.readSchema [awholynewcollection_0_shard5_replica4] Schema name=test
[junit4:junit4]   2> 2416429 T5056 C1033 P50637 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7dacb2d7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6632e8fd),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7dacb2d7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6632e8fd),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2416438 T5056 C1033 P50637 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2416439 T5056 C1033 P50637 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 2416439 T5056 C1033 P50637 oass.SolrIndexSearcher.<init> Opening Searcher@17a087b main
[junit4:junit4]   2> 2416451 T5052 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher@17a087b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2416454 T5037 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2416455 T5056 C1033 P50637 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/index.20130522001605489 [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/index.20130522001605489;done=true>>]
[junit4:junit4]   2> 2416456 T5056 C1033 P50637 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/index.20130522001605489
[junit4:junit4]   2> 2416456 T5056 C1033 P50637 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369170949813/nodes_used_collection_shard1_replica2/data/index.20130522001605489
[junit4:junit4]   2> 2416458 T5056 C1033 P50637 oasc.RecoveryStrategy.replay No replay needed. core=nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2416460 T5056 C1033 P50637 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2416461 T5056 C1033 P50637 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=active
[junit4:junit4]   2> 2416466 T5056 C1033 P50637 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 2416470 T5037 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2416484 T5037 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica1] Schema name=test
[junit4:junit4]   2> 2416565 T4985 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2416567 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50627__%2Fi_nodes_used_collection_shard2_replica2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50627__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50627/_/i"}
[junit4:junit4]   2> 2416579 T4985 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50637__%2Fi_nodes_used_collection_shard1_replica2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50637__%2Fi",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50637/_/i"}
[junit4:junit4]   2> 2416716 T4989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2416717 T5044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2416717 T5030 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2416718 T5016 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2416719 T5059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2416719 T5061 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2416720 T5002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2416722 T4984 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2416902 T5011 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2416913 T5025 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2416926 T4976 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2416927 T5025 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2416939 T5011 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2416972 T4976 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2416979 T4976 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica5] Schema name=test
[junit4:junit4]   2> 2417010 T5035 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2417028 T5035 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2417103 T5025 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica4] Schema name=test
[junit4:junit4]   2> 2417260 T5035 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica1] Schema name=test
[junit4:junit4]   2> 2417280 T4997 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2417444 T4997 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2417452 T4997 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica3] Schema name=test
[junit4:junit4]   2> 2417478 T5011 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica2] Schema name=test
[junit4:junit4]   2> 2432334 T4978 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2432422 T4978 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2432425 T5007 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2432428 T4978 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2432570 T5038 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2432632 T5007 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2432645 T4978 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2432663 T5007 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2432742 T5080 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2432783 T4998 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2432817 T5007 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2432828 T5080 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2432835 T5080 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2432877 T5038 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2432883 T5038 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2433078 T4978 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2433153 T4998 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2433159 T4998 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2433199 T5038 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2433232 T4978 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2433243 T4978 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2433244 T4978 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2433245 T4978 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2433269 T4978 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2433270 T4978 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 2433270 T4978 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2433271 T4978 oasc.SolrCore.<init> [awholynewcollection_0_shard2_replica5] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard2_replica5/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard2_replica5/data/
[junit4:junit4]   2> 2433272 T4978 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6d56bbcf
[junit4:junit4]   2> 2433272 T4978 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2433278 T4978 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard2_replica5/data
[junit4:junit4]   2> 2433296 T4978 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard2_replica5/data/index/
[junit4:junit4]   2> 2433297 T4978 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard2_replica5] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369170944323/awholynewcollection_0_shard2_replica5/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2433311 T4978 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/

[...truncated too long message...]

!! WARNING: best effort to remove /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369172634571 FAILED !!!!!
[junit4:junit4]   2> 4158365 T7280 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 4158365 T7280 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51320 51320
[junit4:junit4]   1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=32EEEE9975F380AA -Dtests.slow=true -Dtests.locale=es_PE -Dtests.timezone=Europe/Moscow -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR   73.0s | ChaosMonkeyShardSplitTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:51327 returned non ok status:500, message:Server Error
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([32EEEE9975F380AA:B308608102ACE096]:0)
[junit4:junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4]    > 	at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
[junit4:junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 4158398 T7280 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 73059 T7279 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4]   2> 4159395 T7371 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 4159396 T7371 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {id=PostingsFormat(name=Memory doPackFST= false), _version_=PostingsFormat(name=Asserting)}, docValues:{}, sim=DefaultSimilarity, locale=es_PE, timezone=Europe/Moscow
[junit4:junit4]   2> NOTE: Mac OS X 10.8.3 x86_64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=2,threads=3,free=208935408,total=385875968
[junit4:junit4]   2> NOTE: All tests run in this JVM: [EchoParamsTest, ChaosMonkeyNothingIsSafeTest, TestFoldingMultitermQuery, UpdateRequestProcessorFactoryTest, TestFieldTypeResource, TestSolrQueryParser, NoCacheHeaderTest, ShardSplitTest, ChaosMonkeySafeLeaderTest, JSONWriterTest, TestSolrQueryParserDefaultOperatorResource, DirectSolrConnectionTest, SolrCmdDistributorTest, TestDocumentBuilder, SynonymTokenizerTest, CircularListTest, ZkControllerTest, TestDocSet, SpellCheckComponentTest, SignatureUpdateProcessorFactoryTest, SuggesterWFSTTest, TestRTGBase, TestLuceneMatchVersion, TestPseudoReturnFields, PreAnalyzedFieldTest, FastVectorHighlighterTest, TestCoreContainer, UpdateParamsTest, TestSerializedLuceneMatchVersion, TimeZoneUtilsTest, TestPhraseSuggestions, TestSolrDeletionPolicy1, ShardRoutingCustomTest, ClusterStateTest, TestFastOutputStream, CSVRequestHandlerTest, ClusterStateUpdateTest, TestCodecSupport, UniqFieldsUpdateProcessorFactoryTest, CurrencyFieldOpenExchangeTest, TestCSVLoader, HighlighterTest, SolrCoreTest, ZkNodePropsTest, TestHashPartitioner, TestSearchPerf, RequiredFieldsTest, LeaderElectionTest, MinimalSchemaTest, TestXIncludeConfig, SystemInfoHandlerTest, DebugComponentTest, TermVectorComponentDistributedTest, TestLRUCache, TestFastLRUCache, TestJmxIntegration, TestSweetSpotSimilarityFactory, StatsComponentTest, SimpleFacetsTest, DateMathParserTest, TestDynamicFieldResource, IndexReaderFactoryTest, DocValuesTest, ZkSolrClientTest, TestPropInject, TestDefaultSimilarityFactory, TestMaxScoreQueryParser, TestUniqueKeyFieldResource, FieldAnalysisRequestHandlerTest, TestReversedWildcardFilterFactory, DistanceFunctionTest, SoftAutoCommitTest, DocValuesMultiTest, TestDFRSimilarityFactory, TestMergePolicyConfig, TestManagedSchema, CachingDirectoryFactoryTest, TestStressLucene, TestRandomDVFaceting, TestFieldResource, MoreLikeThisHandlerTest, TestSolrCoreProperties, TestConfig, QueryElevationComponentTest, DistributedQueryElevationComponentTest, TestPerFieldSimilarity, FileBasedSpellCheckerTest, TestDistributedGrouping, FullSolrCloudDistribCmdsTest, BinaryUpdateRequestHandlerTest, TestBadConfig, DistributedSpellCheckComponentTest, QueryResultKeyTest, TestDefaultSearchFieldResource, PathHierarchyTokenizerFactoryTest, TestIndexSearcher, SearchHandlerTest, PeerSyncTest, DirectUpdateHandlerTest, SyncSliceTest, NotRequiredUniqueKeyTest, TestCloudManagedSchemaAddField, ShardRoutingTest, SolrCoreCheckLockOnStartupTest, TestLFUCache, TestSurroundQueryParser, DateFieldTest, AlternateDirectoryTest, StandardRequestHandlerTest, TestOmitPositions, TestFunctionQuery, IndexBasedSpellCheckerTest, ScriptEngineTest, TestFaceting, OpenCloseCoreStressTest, TestRandomFaceting, DocumentAnalysisRequestHandlerTest, ReturnFieldsTest, TestReplicationHandler, TestSolrXml, TestAnalyzedSuggestions, TestLazyCores, TestRealTimeGet, SpellCheckCollatorTest, PluginInfoTest, TestQuerySenderNoQuery, OverseerCollectionProcessorTest, NumericFieldsTest, ZkCLITest, TestAddFieldRealTimeGet, TestPropInjectDefaults, HardAutoCommitTest, BadCopyFieldTest, TestCollationField, BasicDistributedZk2Test, SolrPluginUtilsTest, TestUtils, LukeRequestHandlerTest, TestIndexingPerformance, PrimUtilsTest, CacheHeaderTest, BasicDistributedZkTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, BasicZkTest, RecoveryZkTest, TestZkChroot, SliceStateUpdateTest, TestDistributedSearch, AutoCommitTest, TestRecovery, TestStressReorder, TestJoin, TestMultiCoreConfBootstrap, TestReload, TestStressVersions, DistributedTermsComponentTest, TestRangeQuery, TestSolr4Spatial, TestGroupingSearch, BadIndexSchemaTest, ConvertedLegacyTest, TestSort, TestFiltering, BasicFunctionalityTest, ShowFileRequestHandlerTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, SuggesterTSTTest, SuggesterTest, TestTrie, SpatialFilterTest, PolyFieldTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestUpdate, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, TestRemoteStreaming, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, DisMaxRequestHandlerTest, TestQueryUtils, TestWriterPerf, DirectSolrSpellCheckerTest, TestQueryTypes, PrimitiveFieldTypeTest, TermsComponentTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, DocumentBuilderTest, TestValueSourceCache, TermVectorComponentTest, TestArbitraryIndexDir, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, RegexBoostProcessorTest, IndexSchemaTest, TestCSVResponseWriter, MBeansHandlerTest, QueryParsingTest, JsonLoaderTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, PingRequestHandlerTest, TestComponentsName, CoreContainerCoreInitFailuresTest, TestBinaryResponseWriter, HighlighterConfigTest, SOLR749Test, TestQuerySenderListener, TestSolrIndexConfig, CopyFieldTest, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, TestSolrDeletionPolicy2, MultiTermTest, SampleTest, TestBinaryField, TestElisionMultitermQuery, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, SpellPossibilityIteratorTest, TestCharFilters, TestNumberUtils, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, TestPluginEnable, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, TestSolrXMLSerializer, TestSystemIdResolver, TestSuggestSpellingConverter, SpellingQueryConverterTest, DOMUtilTest, RAMDirectoryFactoryTest, TestSolrJ, SliceStateTest, UUIDFieldTest, FileUtilsTest, SolrTestCaseJ4Test, AliasIntegrationTest, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed in 74.13s, 1 test, 1 error <<< FAILURES!

[...truncated 52 lines...]
BUILD FAILED
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:380: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:360: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:181: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:437: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1240: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:884: There were test failures: 295 suites, 1231 tests, 1 error, 1 failure, 18 ignored (6 assumptions)

Total time: 122 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 494 - Still Failing!

Posted by Michael McCandless <lu...@mikemccandless.com>.
I opened https://issues.apache.org/jira/browse/LUCENE-5017

Mike McCandless

http://blog.mikemccandless.com


On Thu, May 23, 2013 at 11:44 AM, Robert Muir <rm...@apache.org> wrote:
> Is anyone looking into these? this test has been failing several times and
> reproduces exactly:
>
> rmuir@beast:~/workspace/lucene-trunk/lucene/spatial$ ant test
> -Dtests.seed=456022665217DADF -Dtests.jvms=1
>
>
>
> On Wed, May 22, 2013 at 8:58 PM, Policeman Jenkins Server
> <je...@thetaphi.de> wrote:
>>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/494/
>> Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
>>
>> 1 tests failed.
>> FAILED:
>> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains
>> {#3 seed=[456022665217DADF:2C2A2816BD2BA1C5]}
>>
>> Error Message:
>> Shouldn't match I
>> #0:ShapePair(Rect(minX=207.0,maxX=213.0,minY=-108.0,maxY=4.0) ,
>> Rect(minX=4.0,maxX=11.0,minY=-76.0,maxY=122.0))
>> Q:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
>>
>> Stack Trace:
>> java.lang.AssertionError: Shouldn't match I
>> #0:ShapePair(Rect(minX=207.0,maxX=213.0,minY=-108.0,maxY=4.0) ,
>> Rect(minX=4.0,maxX=11.0,minY=-76.0,maxY=122.0))
>> Q:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
>>         at
>> __randomizedtesting.SeedInfo.seed([456022665217DADF:2C2A2816BD2BA1C5]:0)
>>         at org.junit.Assert.fail(Assert.java:93)
>>         at
>> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:287)
>>         at
>> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:273)
>>         at
>> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains(SpatialOpRecursivePrefixTreeTest.java:101)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:601)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>>         at
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>         at
>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>>         at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>         at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>         at
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>         at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>>         at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>>         at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>>         at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>         at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>         at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>         at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>         at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>>         at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>         at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>>         at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>         at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>         at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>>         at java.lang.Thread.run(Thread.java:722)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 8259 lines...]
>> [junit4:junit4] Suite:
>> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   2>
>> Ig:ShapePair(Rect(minX=128.0,maxX=256.0,minY=-128.0,maxY=128.0) ,
>> Rect(minX=0.0,maxX=128.0,minY=-128.0,maxY=128.0))
>> Qg:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
>> [junit4:junit4]   2> NOTE: reproduce with: ant test
>> -Dtestcase=SpatialOpRecursivePrefixTreeTest -Dtests.method="testContains {#3
>> seed=[456022665217DADF:2C2A2816BD2BA1C5]}" -Dtests.seed=456022665217DADF
>> -Dtests.slow=true -Dtests.locale=nl_BE -Dtests.timezone=Poland
>> -Dtests.file.encoding=ISO-8859-1
>> [junit4:junit4] FAILURE 0.08s |
>> SpatialOpRecursivePrefixTreeTest.testContains {#3
>> seed=[456022665217DADF:2C2A2816BD2BA1C5]} <<<
>> [junit4:junit4]    > Throwable #1: java.lang.AssertionError: Shouldn't
>> match I #0:ShapePair(Rect(minX=207.0,maxX=213.0,minY=-108.0,maxY=4.0) ,
>> Rect(minX=4.0,maxX=11.0,minY=-76.0,maxY=122.0))
>> Q:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
>> [junit4:junit4]    >    at
>> __randomizedtesting.SeedInfo.seed([456022665217DADF:2C2A2816BD2BA1C5]:0)
>> [junit4:junit4]    >    at
>> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:287)
>> [junit4:junit4]    >    at
>> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:273)
>> [junit4:junit4]    >    at
>> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains(SpatialOpRecursivePrefixTreeTest.java:101)
>> [junit4:junit4]    >    at java.lang.Thread.run(Thread.java:722)
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   1> Strategy:
>> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
>> calculator=CartesianDistCalc,
>> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>> [junit4:junit4]   2> NOTE: test params are: codec=Lucene41,
>> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=nl_BE,
>> timezone=Poland
>> [junit4:junit4]   2> NOTE: Mac OS X 10.8.3 x86_64/Oracle Corporation
>> 1.7.0_21 (64-bit)/cpus=2,threads=1,free=42505272,total=93585408
>> [junit4:junit4]   2> NOTE: All tests run in this JVM:
>> [TestTermQueryPrefixGridStrategy, TestBBoxStrategy, DistanceStrategyTest,
>> JtsPolygonTest, SpatialOpRecursivePrefixTreeTest]
>> [junit4:junit4] Completed in 8.18s, 42 tests, 1 failure <<< FAILURES!
>>
>> [...truncated 33 lines...]
>> BUILD FAILED
>>
>> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:380:
>> The following error occurred while executing this line:
>>
>> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:360:
>> The following error occurred while executing this line:
>>
>> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39:
>> The following error occurred while executing this line:
>>
>> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/build.xml:554:
>> The following error occurred while executing this line:
>>
>> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1825:
>> The following error occurred while executing this line:
>>
>> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/module-build.xml:61:
>> The following error occurred while executing this line:
>>
>> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1240:
>> The following error occurred while executing this line:
>>
>> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:884:
>> There were test failures: 13 suites, 74 tests, 1 failure, 2 ignored (2
>> assumptions)
>>
>> Total time: 46 minutes 38 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops
>> -XX:+UseSerialGC
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 494 - Still Failing!

Posted by Robert Muir <rm...@apache.org>.
Is anyone looking into these? this test has been failing several times and
reproduces exactly:

rmuir@beast:~/workspace/lucene-trunk/lucene/spatial$ ant test
-Dtests.seed=456022665217DADF -Dtests.jvms=1


On Wed, May 22, 2013 at 8:58 PM, Policeman Jenkins Server <
jenkins@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/494/
> Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
>
> 1 tests failed.
> FAILED:
>  org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains
> {#3 seed=[456022665217DADF:2C2A2816BD2BA1C5]}
>
> Error Message:
> Shouldn't match I
> #0:ShapePair(Rect(minX=207.0,maxX=213.0,minY=-108.0,maxY=4.0) ,
> Rect(minX=4.0,maxX=11.0,minY=-76.0,maxY=122.0))
> Q:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
>
> Stack Trace:
> java.lang.AssertionError: Shouldn't match I
> #0:ShapePair(Rect(minX=207.0,maxX=213.0,minY=-108.0,maxY=4.0) ,
> Rect(minX=4.0,maxX=11.0,minY=-76.0,maxY=122.0))
> Q:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
>         at
> __randomizedtesting.SeedInfo.seed([456022665217DADF:2C2A2816BD2BA1C5]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at
> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:287)
>         at
> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:273)
>         at
> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains(SpatialOpRecursivePrefixTreeTest.java:101)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at java.lang.Thread.run(Thread.java:722)
>
>
>
>
> Build Log:
> [...truncated 8259 lines...]
> [junit4:junit4] Suite:
> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   2>
> Ig:ShapePair(Rect(minX=128.0,maxX=256.0,minY=-128.0,maxY=128.0) ,
> Rect(minX=0.0,maxX=128.0,minY=-128.0,maxY=128.0))
> Qg:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
> [junit4:junit4]   2> NOTE: reproduce with: ant test
>  -Dtestcase=SpatialOpRecursivePrefixTreeTest -Dtests.method="testContains
> {#3 seed=[456022665217DADF:2C2A2816BD2BA1C5]}"
> -Dtests.seed=456022665217DADF -Dtests.slow=true -Dtests.locale=nl_BE
> -Dtests.timezone=Poland -Dtests.file.encoding=ISO-8859-1
> [junit4:junit4] FAILURE 0.08s |
> SpatialOpRecursivePrefixTreeTest.testContains {#3
> seed=[456022665217DADF:2C2A2816BD2BA1C5]} <<<
> [junit4:junit4]    > Throwable #1: java.lang.AssertionError: Shouldn't
> match I #0:ShapePair(Rect(minX=207.0,maxX=213.0,minY=-108.0,maxY=4.0) ,
> Rect(minX=4.0,maxX=11.0,minY=-76.0,maxY=122.0))
> Q:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
> [junit4:junit4]    >    at
> __randomizedtesting.SeedInfo.seed([456022665217DADF:2C2A2816BD2BA1C5]:0)
> [junit4:junit4]    >    at
> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:287)
> [junit4:junit4]    >    at
> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:273)
> [junit4:junit4]    >    at
> org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains(SpatialOpRecursivePrefixTreeTest.java:101)
> [junit4:junit4]    >    at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   1> Strategy:
> RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false,
> calculator=CartesianDistCalc,
> worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> [junit4:junit4]   2> NOTE: test params are: codec=Lucene41,
> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=nl_BE,
> timezone=Poland
> [junit4:junit4]   2> NOTE: Mac OS X 10.8.3 x86_64/Oracle Corporation
> 1.7.0_21 (64-bit)/cpus=2,threads=1,free=42505272,total=93585408
> [junit4:junit4]   2> NOTE: All tests run in this JVM:
> [TestTermQueryPrefixGridStrategy, TestBBoxStrategy, DistanceStrategyTest,
> JtsPolygonTest, SpatialOpRecursivePrefixTreeTest]
> [junit4:junit4] Completed in 8.18s, 42 tests, 1 failure <<< FAILURES!
>
> [...truncated 33 lines...]
> BUILD FAILED
> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:380:
> The following error occurred while executing this line:
> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:360:
> The following error occurred while executing this line:
> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39:
> The following error occurred while executing this line:
> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/build.xml:554:
> The following error occurred while executing this line:
> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1825:
> The following error occurred while executing this line:
> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/module-build.xml:61:
> The following error occurred while executing this line:
> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1240:
> The following error occurred while executing this line:
> /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:884:
> There were test failures: 13 suites, 74 tests, 1 failure, 2 ignored (2
> assumptions)
>
> Total time: 46 minutes 38 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops
> -XX:+UseSerialGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 494 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/494/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains {#3 seed=[456022665217DADF:2C2A2816BD2BA1C5]}

Error Message:
Shouldn't match I #0:ShapePair(Rect(minX=207.0,maxX=213.0,minY=-108.0,maxY=4.0) , Rect(minX=4.0,maxX=11.0,minY=-76.0,maxY=122.0)) Q:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)

Stack Trace:
java.lang.AssertionError: Shouldn't match I #0:ShapePair(Rect(minX=207.0,maxX=213.0,minY=-108.0,maxY=4.0) , Rect(minX=4.0,maxX=11.0,minY=-76.0,maxY=122.0)) Q:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
	at __randomizedtesting.SeedInfo.seed([456022665217DADF:2C2A2816BD2BA1C5]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:287)
	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:273)
	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains(SpatialOpRecursivePrefixTreeTest.java:101)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 8259 lines...]
[junit4:junit4] Suite: org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   2> Ig:ShapePair(Rect(minX=128.0,maxX=256.0,minY=-128.0,maxY=128.0) , Rect(minX=0.0,maxX=128.0,minY=-128.0,maxY=128.0)) Qg:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SpatialOpRecursivePrefixTreeTest -Dtests.method="testContains {#3 seed=[456022665217DADF:2C2A2816BD2BA1C5]}" -Dtests.seed=456022665217DADF -Dtests.slow=true -Dtests.locale=nl_BE -Dtests.timezone=Poland -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 0.08s | SpatialOpRecursivePrefixTreeTest.testContains {#3 seed=[456022665217DADF:2C2A2816BD2BA1C5]} <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: Shouldn't match I #0:ShapePair(Rect(minX=207.0,maxX=213.0,minY=-108.0,maxY=4.0) , Rect(minX=4.0,maxX=11.0,minY=-76.0,maxY=122.0)) Q:Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([456022665217DADF:2C2A2816BD2BA1C5]:0)
[junit4:junit4]    > 	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:287)
[junit4:junit4]    > 	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:273)
[junit4:junit4]    > 	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains(SpatialOpRecursivePrefixTreeTest.java:101)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=nl_BE, timezone=Poland
[junit4:junit4]   2> NOTE: Mac OS X 10.8.3 x86_64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=2,threads=1,free=42505272,total=93585408
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestTermQueryPrefixGridStrategy, TestBBoxStrategy, DistanceStrategyTest, JtsPolygonTest, SpatialOpRecursivePrefixTreeTest]
[junit4:junit4] Completed in 8.18s, 42 tests, 1 failure <<< FAILURES!

[...truncated 33 lines...]
BUILD FAILED
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:380: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:360: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/build.xml:554: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1825: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/module-build.xml:61: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1240: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:884: There were test failures: 13 suites, 74 tests, 1 failure, 2 ignored (2 assumptions)

Total time: 46 minutes 38 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 493 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/493/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
collection already exists: awholynewcollection_1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: collection already exists: awholynewcollection_1
	at __randomizedtesting.SeedInfo.seed([12BA1200442C22A9:935C9C1833734295]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:424)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:264)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:306)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:438)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:146)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 9216 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4:junit4]   2> 415061 T762 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /n_agm/
[junit4:junit4]   2> 415072 T762 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1369215565921
[junit4:junit4]   2> 415074 T762 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 415075 T763 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 415190 T762 oasc.ZkTestServer.run start zk server on port:49407
[junit4:junit4]   2> 415218 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 415230 T769 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@128936ff name:ZooKeeperConnection Watcher:127.0.0.1:49407 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 415235 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 415235 T762 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 415311 T771 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fe39445 name:ZooKeeperConnection Watcher:127.0.0.1:49407/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 415311 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 415313 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 415320 T762 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 415380 T762 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 415396 T762 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 415408 T762 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 415419 T762 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 415420 T762 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 415435 T762 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 415437 T762 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 415452 T762 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 415453 T762 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 415463 T762 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 415464 T762 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 415473 T762 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 415474 T762 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 415486 T762 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 415487 T762 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 415497 T762 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 415498 T762 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 415513 T762 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 415514 T762 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 415527 T762 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 415528 T762 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 416100 T762 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 416116 T762 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49410
[junit4:junit4]   2> 416117 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 416117 T762 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 416118 T762 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388
[junit4:junit4]   2> 416118 T762 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/solr.xml
[junit4:junit4]   2> 416119 T762 oasc.CoreContainer.<init> New CoreContainer 1493653053
[junit4:junit4]   2> 416120 T762 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/'
[junit4:junit4]   2> 416120 T762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/'
[junit4:junit4]   2> 416309 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388
[junit4:junit4]   2> 416311 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/collection1
[junit4:junit4]   2> 416313 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/collection1/conf
[junit4:junit4]   2> 416484 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/collection1/conf/xslt
[junit4:junit4]   2> 416489 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/collection1/lib
[junit4:junit4]   2> 416490 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/collection1/lib/classes
[junit4:junit4]   2> 416494 T762 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/conf/core.properties, adding to cores
[junit4:junit4]   2> 416518 T762 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 416519 T762 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 416520 T762 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 416521 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 416521 T762 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 416522 T762 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 416522 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 416523 T762 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 416523 T762 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 416524 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 416554 T762 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 416554 T762 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49407/solr
[junit4:junit4]   2> 416555 T762 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 416559 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 416565 T782 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7347e756 name:ZooKeeperConnection Watcher:127.0.0.1:49407 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 416566 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 416581 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 416604 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 416610 T784 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a3dba50 name:ZooKeeperConnection Watcher:127.0.0.1:49407/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 416610 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 416615 T762 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 416625 T762 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 416637 T762 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 416649 T762 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49410_n_agm
[junit4:junit4]   2> 416652 T762 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49410_n_agm
[junit4:junit4]   2> 416662 T762 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 416678 T762 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 416687 T762 oasc.Overseer.start Overseer (id=89732911329640451-127.0.0.1:49410_n_agm-n_0000000000) starting
[junit4:junit4]   2> 416697 T762 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 416714 T786 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 416717 T762 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 416728 T762 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 416733 T762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 416745 T762 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 416746 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 416747 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 416762 T785 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 416784 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 416791 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 416795 T789 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9ae71e name:ZooKeeperConnection Watcher:127.0.0.1:49407/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 416796 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 416800 T762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 416811 T762 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 417432 T762 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 417445 T762 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49414
[junit4:junit4]   2> 417446 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 417447 T762 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 417447 T762 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659
[junit4:junit4]   2> 417448 T762 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/solr.xml
[junit4:junit4]   2> 417448 T762 oasc.CoreContainer.<init> New CoreContainer 1935210090
[junit4:junit4]   2> 417449 T762 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/'
[junit4:junit4]   2> 417450 T762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/'
[junit4:junit4]   2> 417622 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659
[junit4:junit4]   2> 417624 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/collection1
[junit4:junit4]   2> 417626 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/collection1/conf
[junit4:junit4]   2> 417813 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/collection1/conf/xslt
[junit4:junit4]   2> 417819 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/collection1/lib
[junit4:junit4]   2> 417821 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/collection1/lib/classes
[junit4:junit4]   2> 417825 T762 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/conf/core.properties, adding to cores
[junit4:junit4]   2> 417851 T762 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 417852 T762 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 417853 T762 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 417853 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 417854 T762 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 417855 T762 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 417855 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 417856 T762 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 417856 T762 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 417857 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 417886 T762 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 417887 T762 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49407/solr
[junit4:junit4]   2> 417887 T762 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 417890 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 417895 T800 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b05d6a0 name:ZooKeeperConnection Watcher:127.0.0.1:49407 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 417897 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 417903 T764 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13ecb9b80770005, likely client has closed socket
[junit4:junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 417908 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 417938 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 417943 T802 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d7e633b name:ZooKeeperConnection Watcher:127.0.0.1:49407/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 417943 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 417959 T762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 418982 T762 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49414_n_agm
[junit4:junit4]   2> 418985 T762 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49414_n_agm
[junit4:junit4]   2> 418994 T789 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 418996 T784 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 418999 T802 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 419024 T762 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 419025 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 419026 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 419641 T762 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 419652 T762 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49417
[junit4:junit4]   2> 419653 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 419653 T762 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 419654 T762 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899
[junit4:junit4]   2> 419655 T762 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/solr.xml
[junit4:junit4]   2> 419656 T762 oasc.CoreContainer.<init> New CoreContainer 1445004885
[junit4:junit4]   2> 419656 T762 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/'
[junit4:junit4]   2> 419657 T762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/'
[junit4:junit4]   2> 419849 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899
[junit4:junit4]   2> 419851 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/collection1
[junit4:junit4]   2> 419853 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/collection1/conf
[junit4:junit4]   2> 420056 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/collection1/conf/xslt
[junit4:junit4]   2> 420061 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/collection1/lib
[junit4:junit4]   2> 420063 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/collection1/lib/classes
[junit4:junit4]   2> 420069 T762 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/conf/core.properties, adding to cores
[junit4:junit4]   2> 420093 T762 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 420093 T762 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 420094 T762 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 420095 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 420095 T762 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 420096 T762 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 420097 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 420097 T762 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 420098 T762 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 420099 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 420129 T762 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 420130 T762 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49407/solr
[junit4:junit4]   2> 420130 T762 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 420134 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 420492 T814 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@545d315a name:ZooKeeperConnection Watcher:127.0.0.1:49407 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 420493 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 420500 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 420529 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 420553 T816 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7060388c name:ZooKeeperConnection Watcher:127.0.0.1:49407/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 420553 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 420570 T762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 421592 T762 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49417_n_agm
[junit4:junit4]   2> 421601 T762 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49417_n_agm
[junit4:junit4]   2> 421613 T802 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 421614 T784 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 421615 T789 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 421623 T816 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 421638 T762 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 421639 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 421639 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 422263 T762 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 422271 T762 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49420
[junit4:junit4]   2> 422272 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 422273 T762 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 422274 T762 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514
[junit4:junit4]   2> 422274 T762 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/solr.xml
[junit4:junit4]   2> 422275 T762 oasc.CoreContainer.<init> New CoreContainer 914212025
[junit4:junit4]   2> 422296 T762 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/'
[junit4:junit4]   2> 422297 T762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/'
[junit4:junit4]   2> 422466 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514
[junit4:junit4]   2> 422469 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/collection1
[junit4:junit4]   2> 422471 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/collection1/conf
[junit4:junit4]   2> 422655 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/collection1/conf/xslt
[junit4:junit4]   2> 422661 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/collection1/lib
[junit4:junit4]   2> 422663 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/collection1/lib/classes
[junit4:junit4]   2> 422666 T762 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/conf/core.properties, adding to cores
[junit4:junit4]   2> 422692 T762 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 422693 T762 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 422694 T762 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 422695 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 422695 T762 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 422696 T762 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 422697 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 422698 T762 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 422698 T762 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 422699 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 422731 T762 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 422733 T762 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49407/solr
[junit4:junit4]   2> 422733 T762 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 422736 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 422742 T828 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64d783dd name:ZooKeeperConnection Watcher:127.0.0.1:49407 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 422743 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 422754 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 422783 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 422790 T830 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cc8f74a name:ZooKeeperConnection Watcher:127.0.0.1:49407/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 422791 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 422813 T762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 423828 T762 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49420_n_agm
[junit4:junit4]   2> 423831 T762 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49420_n_agm
[junit4:junit4]   2> 423851 T830 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 423851 T789 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 423851 T816 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 423852 T784 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 423852 T802 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 423865 T762 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 423866 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 423867 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 424545 T762 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 424556 T762 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49423
[junit4:junit4]   2> 424558 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 424559 T762 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 424561 T762 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745
[junit4:junit4]   2> 424562 T762 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/solr.xml
[junit4:junit4]   2> 424562 T762 oasc.CoreContainer.<init> New CoreContainer 282132524
[junit4:junit4]   2> 424563 T762 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/'
[junit4:junit4]   2> 424564 T762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/'
[junit4:junit4]   2> 424739 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745
[junit4:junit4]   2> 424741 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/collection1
[junit4:junit4]   2> 424743 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/collection1/conf
[junit4:junit4]   2> 424923 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/collection1/conf/xslt
[junit4:junit4]   2> 424928 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/collection1/lib
[junit4:junit4]   2> 424930 T762 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/collection1/lib/classes
[junit4:junit4]   2> 424933 T762 oasc.SolrCoreDiscoverer.addCore Discovered properties file /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/conf/core.properties, adding to cores
[junit4:junit4]   2> 424948 T762 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 424949 T762 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 424950 T762 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 424951 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 424951 T762 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 424952 T762 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 424952 T762 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 424953 T762 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 424954 T762 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 424954 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 424986 T762 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 424987 T762 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49407/solr
[junit4:junit4]   2> 424988 T762 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 424992 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 424997 T842 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45af66e0 name:ZooKeeperConnection Watcher:127.0.0.1:49407 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 424998 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 425003 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 425029 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 425035 T844 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ac330aa name:ZooKeeperConnection Watcher:127.0.0.1:49407/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 425038 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 425055 T762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 426069 T762 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49423_n_agm
[junit4:junit4]   2> 426076 T762 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49423_n_agm
[junit4:junit4]   2> 426088 T784 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 426091 T816 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 426091 T844 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 426091 T789 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 426093 T830 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 426093 T802 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 426105 T762 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 426105 T762 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 426106 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 426141 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 426177 T796 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2
[junit4:junit4]   2> 426185 T784 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 426190 T786 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
[junit4:junit4]   2> 	  "operation":"createcollection",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "maxShardsPerNode":null,
[junit4:junit4]   2> 	  "createNodeSet":null,
[junit4:junit4]   2> 	  "name":"nodes_used_collection",
[junit4:junit4]   2> 	  "replicationFactor":"2"}
[junit4:junit4]   2> 426191 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:49423_n_agm
[junit4:junit4]   2> 426192 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:49420_n_agm
[junit4:junit4]   2> 426194 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:49410_n_agm
[junit4:junit4]   2> 426195 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:49417_n_agm
[junit4:junit4]   2> 426206 T824 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 426206 T824 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 426219 T824 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 426221 T824 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 426222 T811 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 426221 T840 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 426223 T840 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 426223 T811 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 426222 T775 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 426225 T775 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4]   2> 426226 T824 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 426227 T840 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 426227 T840 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 426227 T824 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 426229 T811 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 426229 T811 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 426229 T775 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4]   2> 426231 T775 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 426235 T840 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 426235 T840 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 426237 T811 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 426239 T811 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 426241 T775 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 426241 T775 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4]   2> 426245 T824 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 426250 T824 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/nodes_used_collection_shard1_replica2/'
[junit4:junit4]   2> 426285 T811 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 426286 T840 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 426287 T775 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4]   2> 426293 T811 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/nodes_used_collection_shard2_replica2/'
[junit4:junit4]   2> 426293 T775 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/'
[junit4:junit4]   2> 426297 T840 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/'
[junit4:junit4]   2> 426536 T824 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 426561 T811 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 426589 T840 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 426643 T775 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 426867 T840 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 426894 T811 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 426897 T840 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 426902 T811 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 426903 T824 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 426908 T824 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 426920 T824 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
[junit4:junit4]   2> 426941 T840 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
[junit4:junit4]   2> 426950 T811 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
[junit4:junit4]   2> 426974 T775 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 426999 T775 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 427008 T775 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
[junit4:junit4]   2> 430213 T824 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 430256 T824 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 430265 T824 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 430277 T811 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 430310 T840 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 430329 T811 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 430335 T811 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 430357 T840 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 430390 T840 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 430452 T775 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 430477 T824 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 430497 T824 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 430500 T811 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 430509 T824 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 430513 T824 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 430515 T824 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 430515 T824 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 430523 T824 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 430524 T824 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 430525 T824 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 430526 T824 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/nodes_used_collection_shard1_replica2/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/nodes_used_collection_shard1_replica2/data/
[junit4:junit4]   2> 430526 T824 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15858f02
[junit4:junit4]   2> 430528 T824 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 430530 T811 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 430533 T840 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 430540 T811 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 430542 T824 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/nodes_used_collection_shard1_replica2/data
[junit4:junit4]   2> 430546 T775 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 430551 T775 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 430553 T824 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/nodes_used_collection_shard1_replica2/data/index/
[junit4:junit4]   2> 430566 T824 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 430565 T811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 430571 T811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 430572 T811 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 430569 T824 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/nodes_used_collection_shard1_replica2/data/index
[junit4:junit4]   2> 430572 T840 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 430584 T824 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@116023a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7bf47635),segFN=segments_1,generation=1}
[junit4:junit4]   2> 430585 T824 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 430598 T824 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 430599 T824 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 430599 T811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 430610 T811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 430611 T811 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 430611 T811 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/nodes_used_collection_shard2_replica2/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/nodes_used_collection_shard2_replica2/data/
[junit4:junit4]   2> 430615 T824 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 430631 T824 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 430632 T824 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 430633 T824 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 430634 T824 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 430635 T824 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 430636 T824 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 430627 T811 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15858f02
[junit4:junit4]   2> 430623 T840 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 430647 T811 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 430650 T811 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/nodes_used_collection_shard2_replica2/data
[junit4:junit4]   2> 430661 T811 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/nodes_used_collection_shard2_replica2/data/index/
[junit4:junit4]   2> 430662 T811 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 430668 T840 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 430669 T840 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 430669 T840 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 430678 T811 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/nodes_used_collection_shard2_replica2/data/index
[junit4:junit4]   2> 430700 T840 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 430704 T840 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 430709 T840 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 430710 T840 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/
[junit4:junit4]   2> 430715 T840 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15858f02
[junit4:junit4]   2> 430716 T840 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 430713 T811 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@47ed62e1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bd83d66),segFN=segments_1,generation=1}
[junit4:junit4]   2> 430718 T811 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 430718 T824 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 430723 T811 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 430724 T811 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 430725 T811 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 430726 T811 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 430728 T811 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 430729 T811 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 430730 T811 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 430732 T811 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 430733 T811 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 430733 T840 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data
[junit4:junit4]   2> 430734 T840 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/index/
[junit4:junit4]   2> 430735 T840 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 430736 T840 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/index
[junit4:junit4]   2> 430745 T840 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30b36590 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f0a457b),segFN=segments_1,generation=1}
[junit4:junit4]   2> 430746 T840 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 430752 T840 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 430765 T840 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 430767 T840 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 430768 T840 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 430769 T840 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 430770 T840 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 430771 T840 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 430772 T840 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 430773 T840 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 430780 T811 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 430813 T824 oass.SolrIndexSearcher.<init> Opening Searcher@79eb3322 main
[junit4:junit4]   2> 430814 T824 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/nodes_used_collection_shard1_replica2/data/tlog
[junit4:junit4]   2> 430816 T775 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 430820 T840 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 430831 T824 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 430832 T824 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 430844 T824 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down
[junit4:junit4]   2> 430844 T850 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher@79eb3322 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 430866 T775 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 430875 T775 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 430879 T840 oass.SolrIndexSearcher.<init> Opening Searcher@45278ab0 main
[junit4:junit4]   2> 430880 T840 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/tlog
[junit4:junit4]   2> 430881 T840 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 430882 T840 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 430886 T851 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher@45278ab0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 430890 T840 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down
[junit4:junit4]   2> 430894 T775 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 430895 T775 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 430895 T775 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 430900 T775 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 430900 T775 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 430901 T775 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 430902 T775 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/
[junit4:junit4]   2> 430902 T775 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15858f02
[junit4:junit4]   2> 430904 T775 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 430902 T811 oass.SolrIndexSearcher.<init> Opening Searcher@4fd98fd8 main
[junit4:junit4]   2> 430907 T811 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/nodes_used_collection_shard2_replica2/data/tlog
[junit4:junit4]   2> 430905 T775 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data
[junit4:junit4]   2> 430915 T811 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 430915 T775 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/index/
[junit4:junit4]   2> 430917 T775 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 430916 T811 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 430918 T775 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/index
[junit4:junit4]   2> 430923 T852 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher@4fd98fd8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 430924 T775 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7054d899 lockFactory=org.apache.lucene.store.NativeFSLockFactory@43b23922),segFN=segments_1,generation=1}
[junit4:junit4]   2> 430925 T775 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 430927 T811 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down
[junit4:junit4]   2> 430931 T775 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 430932 T775 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 430934 T775 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 430936 T775 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 430937 T775 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 430938 T775 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 430939 T775 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 430940 T775 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 430941 T775 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 430967 T775 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 431038 T775 oass.SolrIndexSearcher.<init> Opening Searcher@7fad3fb8 main
[junit4:junit4]   2> 431040 T775 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/tlog
[junit4:junit4]   2> 431047 T775 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 431050 T775 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 431055 T853 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher@7fad3fb8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 431057 T775 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down
[junit4:junit4]   2> 431863 T785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 431865 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49420_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49420/n_agm"}
[junit4:junit4]   2> 431865 T785 oasc.Overseer$ClusterStateUpdater.createCollection Create collection nodes_used_collection with numShards 2
[junit4:junit4]   2> 431875 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49423_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49423/n_agm"}
[junit4:junit4]   2> 431888 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49417_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49417/n_agm"}
[junit4:junit4]   2> 431899 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49410_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49410/n_agm"}
[junit4:junit4]   2> 431921 T816 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 431921 T784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 431922 T789 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 431921 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 431921 T802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 431921 T844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 431933 T811 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica2
[junit4:junit4]   2> 431934 T811 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica2 address:http://127.0.0.1:49417/n_agm collection:nodes_used_collection shard:shard2
[junit4:junit4]   2> 431936 T811 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
[junit4:junit4]   2> 431959 T811 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 431968 T811 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=2 found=1 timeoutin=179998
[junit4:junit4]   2> 432066 T775 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 432067 T775 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica1 address:http://127.0.0.1:49410/n_agm collection:nodes_used_collection shard:shard2
[junit4:junit4]   2> 432472 T811 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 432473 T811 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 432474 T811 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2/
[junit4:junit4]   2> 432474 T811 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica2 url=http://127.0.0.1:49417/n_agm START replicas=[http://127.0.0.1:49410/n_agm/nodes_used_collection_shard2_replica1/] nUpdates=100
[junit4:junit4]   2> 432475 T811 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica2 url=http://127.0.0.1:49417/n_agm DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 432476 T811 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4]   2> 432477 T811 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4]   2> 432477 T811 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2/
[junit4:junit4]   2> 432478 T811 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard2
[junit4:junit4]   2> ASYNC  NEW_CORE C499 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore@5472d6dc url=http://127.0.0.1:49410/n_agm/nodes_used_collection_shard2_replica1 node=127.0.0.1:49410_n_agm C499_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=down, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49410_n_agm, base_url=http://127.0.0.1:49410/n_agm}
[junit4:junit4]   2> 432507 T780 C499 P49410 oasc.SolrCore.execute [nodes_used_collection_shard2_replica1] webapp=/n_agm path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 432861 T824 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica2
[junit4:junit4]   2> 432861 T824 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica2 address:http://127.0.0.1:49420/n_agm collection:nodes_used_collection shard:shard1
[junit4:junit4]   2> 432863 T824 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard1/election
[junit4:junit4]   2> 432884 T824 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 432892 T824 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=2 found=1 timeoutin=179999
[junit4:junit4]   2> 432899 T840 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 432901 T840 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica1 address:http://127.0.0.1:49423/n_agm collection:nodes_used_collection shard:shard1
[junit4:junit4]   2> 433395 T824 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 433396 T824 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 433399 T824 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2/
[junit4:junit4]   2> 433403 T824 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica2 url=http://127.0.0.1:49420/n_agm START replicas=[http://127.0.0.1:49423/n_agm/nodes_used_collection_shard1_replica1/] nUpdates=100
[junit4:junit4]   2> 433405 T824 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica2 url=http://127.0.0.1:49420/n_agm DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 433405 T824 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4]   2> 433406 T824 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4]   2> 433406 T824 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2/
[junit4:junit4]   2> 433407 T824 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard1
[junit4:junit4]   2> ASYNC  NEW_CORE C500 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore@14eb7c05 url=http://127.0.0.1:49423/n_agm/nodes_used_collection_shard1_replica1 node=127.0.0.1:49423_n_agm C500_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=down, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49423_n_agm, base_url=http://127.0.0.1:49423/n_agm}
[junit4:junit4]   2> 433423 T836 C500 P49423 oasc.SolrCore.execute [nodes_used_collection_shard1_replica1] webapp=/n_agm path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 433493 T785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 433548 T784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 433549 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 433550 T789 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 433550 T802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 433552 T816 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 433554 T844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 433580 T811 oasc.ZkController.register We are http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2/ and leader is http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2/
[junit4:junit4]   2> 433583 T811 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica2 baseURL=http://127.0.0.1:49417/n_agm
[junit4:junit4]   2> 433583 T811 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 433584 T811 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=active
[junit4:junit4]   2> 433590 T811 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 433593 T811 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=7371 
[junit4:junit4]   2> 433597 T824 oasc.ZkController.register We are http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2/ and leader is http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2/
[junit4:junit4]   2> 433597 T824 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica2 baseURL=http://127.0.0.1:49420/n_agm
[junit4:junit4]   2> 433598 T824 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 433599 T824 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=active
[junit4:junit4]   2> 433602 T824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 433605 T824 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=7399 
[junit4:junit4]   2> 433648 T775 oasc.ZkController.register We are http://127.0.0.1:49410/n_agm/nodes_used_collection_shard2_replica1/ and leader is http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2/
[junit4:junit4]   2> 433649 T775 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica1 baseURL=http://127.0.0.1:49410/n_agm
[junit4:junit4]   2> 433650 T775 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 433650 T775 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> 433652 T856 C499 P49410 oasc.RecoveryStrategy.run Starting recovery process.  core=nodes_used_collection_shard2_replica1 recoveringAfterStartup=true
[junit4:junit4]   2> 433653 T856 C499 P49410 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 433654 T775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 433655 T856 C499 P49410 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=recovering
[junit4:junit4]   2> 433657 T775 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=7435 
[junit4:junit4]   2> 433661 T856 C499 P49410 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 433695 T809 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:49410_n_agm_nodes_used_collection_shard2_replica1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 433922 T840 oasc.ZkController.register We are http://127.0.0.1:49423/n_agm/nodes_used_collection_shard1_replica1/ and leader is http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2/
[junit4:junit4]   2> 433923 T840 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica1 baseURL=http://127.0.0.1:49423/n_agm
[junit4:junit4]   2> 433923 T840 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 433924 T840 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> 433925 T857 C500 P49423 oasc.RecoveryStrategy.run Starting recovery process.  core=nodes_used_collection_shard1_replica1 recoveringAfterStartup=true
[junit4:junit4]   2> 433926 T857 C500 P49423 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 433927 T840 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 433927 T857 C500 P49423 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=recovering
[junit4:junit4]   2> 433930 T840 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=7710 
[junit4:junit4]   2> 433934 T857 C500 P49423 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 433934 T786 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: nodes_used_collection
[junit4:junit4]   2> 433985 T821 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:49423_n_agm_nodes_used_collection_shard1_replica1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 433986 T802 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
[junit4:junit4]   2> 433992 T784 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 433993 T786 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=7371},core=nodes_used_collection_shard2_replica2},null={responseHeader={status=0,QTime=7399},core=nodes_used_collection_shard1_replica2},null={responseHeader={status=0,QTime=7435},core=nodes_used_collection_shard2_replica1},null={responseHeader={status=0,QTime=7710},core=nodes_used_collection_shard1_replica1}}}
[junit4:junit4]   2> 434001 T796 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2} status=0 QTime=7824 
[junit4:junit4]   2> 434002 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 434004 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 434009 T859 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63528d0c name:ZooKeeperConnection Watcher:127.0.0.1:49407/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 434010 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 434014 T762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 434022 T762 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 434035 T762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 434036 T861 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11e9e7b5 name:ZooKeeperConnection Watcher:127.0.0.1:49407/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 434036 T762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 434044 T762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 434080 T779 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=4&maxShardsPerNode=4&name=awholynewcollection_0&replicationFactor=4&action=CREATE&wt=javabin&version=2
[junit4:junit4]   2> 434085 T784 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 434090 T784 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4]   2> 434092 T786 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
[junit4:junit4]   2> 	  "operation":"createcollection",
[junit4:junit4]   2> 	  "numShards":"4",
[junit4:junit4]   2> 	  "maxShardsPerNode":"4",
[junit4:junit4]   2> 	  "createNodeSet":null,
[junit4:junit4]   2> 	  "name":"awholynewcollection_0",
[junit4:junit4]   2> 	  "replicationFactor":"4"}
[junit4:junit4]   2> 434092 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica1 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:49417_n_agm
[junit4:junit4]   2> 434094 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica2 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:49423_n_agm
[junit4:junit4]   2> 434099 T838 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_0_shard1_replica2
[junit4:junit4]   2> 434100 T838 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434102 T838 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4]   2> 434103 T838 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 434106 T838 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 434106 T838 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4]   2> 434114 T812 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard1_replica1
[junit4:junit4]   2> 434118 T812 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434118 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica3 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:49414_n_agm
[junit4:junit4]   2> 434121 T812 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434122 T812 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434123 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica4 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:49410_n_agm
[junit4:junit4]   2> 434127 T838 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434129 T795 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica3' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard1_replica3
[junit4:junit4]   2> 434130 T795 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434133 T812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard1_replica1/'
[junit4:junit4]   2> 434134 T795 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434135 T795 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434139 T795 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard1_replica3/'
[junit4:junit4]   2> 434174 T780 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica4' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard1_replica4
[junit4:junit4]   2> 434174 T780 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434133 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica1 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:49420_n_agm
[junit4:junit4]   2> 434227 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica2 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:49417_n_agm
[junit4:junit4]   2> 434230 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica3 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:49423_n_agm
[junit4:junit4]   2> 434235 T839 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica3' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_0_shard2_replica3
[junit4:junit4]   2> 434235 T839 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434237 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard2_replica4 as part of slice shard2 of collection awholynewcollection_0 on 127.0.0.1:49414_n_agm
[junit4:junit4]   2> 434242 T797 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica4' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard2_replica4
[junit4:junit4]   2> 434246 T797 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434249 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica1 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:49410_n_agm
[junit4:junit4]   2> 434254 T780 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434255 T780 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434258 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica2 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:49420_n_agm
[junit4:junit4]   2> 434260 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica3 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:49417_n_agm
[junit4:junit4]   2> 434261 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard3_replica4 as part of slice shard3 of collection awholynewcollection_0 on 127.0.0.1:49423_n_agm
[junit4:junit4]   2> 434254 T807 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard2_replica2
[junit4:junit4]   2> 434268 T807 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434266 T797 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434283 T797 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434267 T776 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard3_replica1
[junit4:junit4]   2> 434284 T776 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434285 T838 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_0_shard1_replica2/'
[junit4:junit4]   2> 434289 T839 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434289 T839 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434274 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica1 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:49414_n_agm
[junit4:junit4]   2> 434274 T836 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica4' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_0_shard3_replica4
[junit4:junit4]   2> 434291 T836 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434273 T807 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434292 T807 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434271 T823 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/awholynewcollection_0_shard3_replica2
[junit4:junit4]   2> 434294 T823 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434270 T808 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard3_replica3' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard3_replica3
[junit4:junit4]   2> 434295 T808 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434297 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica2 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:49410_n_agm
[junit4:junit4]   2> 434301 T823 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434305 T823 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434304 T793 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard4_replica1
[junit4:junit4]   2> 434306 T793 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434305 T808 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434307 T808 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434306 T836 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434308 T836 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434310 T776 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434310 T776 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434312 T793 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434313 T793 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434312 T839 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_0_shard2_replica3/'
[junit4:junit4]   2> 434314 T780 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard1_replica4/'
[junit4:junit4]   2> 434319 T823 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/awholynewcollection_0_shard3_replica2/'
[junit4:junit4]   2> 434328 T797 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard2_replica4/'
[junit4:junit4]   2> 434334 T807 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard2_replica2/'
[junit4:junit4]   2> 434339 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_0_shard3_replica4/'
[junit4:junit4]   2> 434340 T793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard4_replica1/'
[junit4:junit4]   2> 434345 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica3 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:49420_n_agm
[junit4:junit4]   2> 434349 T825 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard2_replica1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/awholynewcollection_0_shard2_replica1
[junit4:junit4]   2> 434353 T825 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434348 T777 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica2' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard4_replica2
[junit4:junit4]   2> 434354 T777 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434355 T776 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard3_replica1/'
[junit4:junit4]   2> 434363 T825 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434364 T825 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434367 T808 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard3_replica3/'
[junit4:junit4]   2> 434369 T777 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434371 T777 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434378 T777 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard4_replica2/'
[junit4:junit4]   2> 434391 T786 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard4_replica4 as part of slice shard4 of collection awholynewcollection_0 on 127.0.0.1:49417_n_agm
[junit4:junit4]   2> 434396 T824 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica3' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/awholynewcollection_0_shard4_replica3
[junit4:junit4]   2> 434398 T824 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434401 T811 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard4_replica4' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard4_replica4
[junit4:junit4]   2> 434401 T811 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4]   2> 434410 T825 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/awholynewcollection_0_shard2_replica1/'
[junit4:junit4]   2> 434421 T811 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434422 T811 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434426 T824 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 434426 T824 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4]   2> 434463 T824 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369215572514/awholynewcollection_0_shard4_replica3/'
[junit4:junit4]   2> 434471 T811 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard4_replica4/'
[junit4:junit4]   2> 434694 T795 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435002 T839 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435034 T838 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435070 T780 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435072 T797 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435165 T785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 435168 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:49417_n_agm_nodes_used_collection_shard2_replica2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49417_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49417/n_agm"}
[junit4:junit4]   2> 435171 T812 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435191 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:49420_n_agm_nodes_used_collection_shard1_replica2",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49420_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49420/n_agm"}
[junit4:junit4]   2> 435204 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:49410_n_agm_nodes_used_collection_shard2_replica1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49410_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49410/n_agm"}
[junit4:junit4]   2> 435220 T825 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435257 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:49423_n_agm_nodes_used_collection_shard1_replica1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49423_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49423/n_agm"}
[junit4:junit4]   2> 435269 T836 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435273 T859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435273 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435274 T861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435274 T816 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435275 T789 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435274 T844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435274 T802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435283 T784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 435304 T793 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435352 T823 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435513 T807 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435533 T777 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435646 T811 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435698 T824 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435703 T809 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:49410_n_agm_nodes_used_collection_shard2_replica1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4]   2> 435704 T809 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:49410_n_agm_nodes_used_collection_shard2_replica1&state=recovering&nodeName=127.0.0.1:49410_n_agm&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard2_replica2&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2009 
[junit4:junit4]   2> 435885 T808 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 435986 T821 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:49423_n_agm_nodes_used_collection_shard1_replica1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4]   2> 435987 T821 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:49423_n_agm_nodes_used_collection_shard1_replica1&state=recovering&nodeName=127.0.0.1:49423_n_agm&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard1_replica2&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002 
[junit4:junit4]   2> 436007 T780 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436095 T780 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436221 T795 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436237 T836 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436245 T776 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 436314 T797 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436347 T795 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436367 T836 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436405 T797 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436412 T780 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica4] Schema name=test
[junit4:junit4]   2> 436427 T797 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica4] Schema name=test
[junit4:junit4]   2> 436484 T836 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica4] Schema name=test
[junit4:junit4]   2> 436495 T795 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica3] Schema name=test
[junit4:junit4]   2> 436542 T838 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436590 T825 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436631 T825 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436639 T825 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica1] Schema name=test
[junit4:junit4]   2> 436653 T807 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436701 T838 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436706 T807 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436713 T839 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436721 T776 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436721 T824 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436767 T776 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436769 T824 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436790 T839 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436802 T839 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica3] Schema name=test
[junit4:junit4]   2> 436819 T838 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica2] Schema name=test
[junit4:junit4]   2> 436858 T777 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 436869 T776 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica1] Schema name=test
[junit4:junit4]   2> 436912 T807 oass.IndexSchema.readSchema [awholynewcollection_0_shard2_replica2] Schema name=test
[junit4:junit4]   2> 436916 T777 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 436950 T824 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica3] Schema name=test
[junit4:junit4]   2> 437006 T793 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 437019 T793 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 437153 T793 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica1] Schema name=test
[junit4:junit4]   2> 437159 T777 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica2] Schema name=test
[junit4:junit4]   2> 437303 T823 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 437321 T823 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 437340 T812 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 437372 T812 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 437380 T812 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica1] Schema name=test
[junit4:junit4]   2> 437683 T823 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica2] Schema name=test
[junit4:junit4]   2> ASYNC  NEW_CORE C501 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore@5472d6dc url=http://127.0.0.1:49410/n_agm/nodes_used_collection_shard2_replica1 node=127.0.0.1:49410_n_agm C501_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49410_n_agm, base_url=http://127.0.0.1:49410/n_agm}
[junit4:junit4]   2> 437720 T856 C501 P49410 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2/ core=nodes_used_collection_shard2_replica1 - recoveringAfterStartup=true
[junit4:junit4]   2> 437723 T856 C501 P49410 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica1 url=http://127.0.0.1:49410/n_agm START replicas=[http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2/] nUpdates=100
[junit4:junit4]   2> 437725 T856 C501 P49410 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 437727 T856 C501 P49410 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 437728 T856 C501 P49410 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 437728 T856 C501 P49410 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 437729 T856 C501 P49410 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 437729 T856 C501 P49410 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2/. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 437730 T856 C501 P49410 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C502 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore@42b2fa23 url=http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2 node=127.0.0.1:49417_n_agm C502_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:49417_n_agm, base_url=http://127.0.0.1:49417/n_agm, leader=true}
[junit4:junit4]   2> 437823 T809 C502 P49417 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/n_agm path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=45 
[junit4:junit4]   2> 437843 T808 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 437865 T808 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 437867 T810 C502 P49417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C503 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore@14eb7c05 url=http://127.0.0.1:49423/n_agm/nodes_used_collection_shard1_replica1 node=127.0.0.1:49423_n_agm C503_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49423_n_agm, base_url=http://127.0.0.1:49423/n_agm}
[junit4:junit4]   2> 438284 T857 C503 P49423 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2/ core=nodes_used_collection_shard1_replica1 - recoveringAfterStartup=true
[junit4:junit4]   2> 438285 T857 C503 P49423 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:49423/n_agm START replicas=[http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2/] nUpdates=100
[junit4:junit4]   2> 438286 T857 C503 P49423 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 438286 T857 C503 P49423 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 438287 T857 C503 P49423 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 438287 T857 C503 P49423 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 438288 T857 C503 P49423 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 438288 T857 C503 P49423 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2/. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 438288 T857 C503 P49423 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C504 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore@42b2fa23 url=http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2 node=127.0.0.1:49417_n_agm C504_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:49417_n_agm, base_url=http://127.0.0.1:49417/n_agm, leader=true}
[junit4:junit4]   2> 438439 T810 C504 P49417 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@47ed62e1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bd83d66),segFN=segments_1,generation=1}
[junit4:junit4]   2> 438474 T810 C504 P49417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 438488 T808 oass.IndexSchema.readSchema [awholynewcollection_0_shard3_replica3] Schema name=test
[junit4:junit4]   2> 438510 T810 C504 P49417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@47ed62e1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bd83d66),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@47ed62e1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bd83d66),segFN=segments_2,generation=2}
[junit4:junit4]   2> 438511 T810 C504 P49417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 438520 T810 C504 P49417 oass.SolrIndexSearcher.<init> Opening Searcher@6c2767 realtime
[junit4:junit4]   2> 438521 T810 C504 P49417 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 438521 T810 C504 P49417 oasup.LogUpdateProcessor.finish [nodes_used_collection_shard2_replica2] webapp=/n_agm path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 654
[junit4:junit4]   2> ASYNC  NEW_CORE C505 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore@5472d6dc url=http://127.0.0.1:49410/n_agm/nodes_used_collection_shard2_replica1 node=127.0.0.1:49410_n_agm C505_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49410_n_agm, base_url=http://127.0.0.1:49410/n_agm}
[junit4:junit4]   2> 438525 T856 C505 P49410 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 438526 T856 C505 P49410 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 438542 T809 C504 P49417 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 438542 T809 C504 P49417 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/n_agm path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=12 
[junit4:junit4]   2> 438544 T856 C505 P49410 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 438545 T856 C505 P49410 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 438545 T856 C505 P49410 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 438584 T810 C504 P49417 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/n_agm path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=5 
[junit4:junit4]   2> 438586 T856 C505 P49410 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 438601 T856 C505 P49410 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/index.20130522053949434
[junit4:junit4]   2> 438601 T856 C505 P49410 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@1e8e2ce1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7898b0e8) fullCopy=false
[junit4:junit4]   2> ASYNC  NEW_CORE C506 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore@42b2fa23 url=http://127.0.0.1:49417/n_agm/nodes_used_collection_shard2_replica2 node=127.0.0.1:49417_n_agm C506_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=active, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:49417_n_agm, base_url=http://127.0.0.1:49417/n_agm, leader=true}
[junit4:junit4]   2> 438628 T809 C506 P49417 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/n_agm path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> ASYNC  NEW_CORE C507 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore@3e5d57b0 url=http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2 node=127.0.0.1:49420_n_agm C507_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=active, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:49420_n_agm, base_url=http://127.0.0.1:49420/n_agm, leader=true}
[junit4:junit4]   2> 438646 T826 C507 P49420 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/n_agm path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> ASYNC  NEW_CORE C508 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore@5472d6dc url=http://127.0.0.1:49410/n_agm/nodes_used_collection_shard2_replica1 node=127.0.0.1:49410_n_agm C508_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49410_n_agm, base_url=http://127.0.0.1:49410/n_agm}
[junit4:junit4]   2> 438677 T856 C508 P49410 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> ASYNC  NEW_CORE C509 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore@5472d6dc url=http://127.0.0.1:49410/n_agm/nodes_used_collection_shard2_replica1 node=127.0.0.1:49410_n_agm C509_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{shard=shard2, state=recovering, core=nodes_used_collection_shard2_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49410_n_agm, base_url=http://127.0.0.1:49410/n_agm}
[junit4:junit4]   2> 438773 T856 C509 P49410 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 438774 T856 C509 P49410 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 438781 T856 C509 P49410 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7054d899 lockFactory=org.apache.lucene.store.NativeFSLockFactory@43b23922),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7054d899 lockFactory=org.apache.lucene.store.NativeFSLockFactory@43b23922),segFN=segments_2,generation=2}
[junit4:junit4]   2> 438781 T856 C509 P49410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 438782 T856 C509 P49410 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 438782 T856 C509 P49410 oass.SolrIndexSearcher.<init> Opening Searcher@24ed3cfa main
[junit4:junit4]   2> 438784 T853 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher@24ed3cfa main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 438841 T856 C509 P49410 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/index.20130522053949434 [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/index.20130522053949434;done=true>>]
[junit4:junit4]   2> 438841 T856 C509 P49410 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/index.20130522053949434
[junit4:junit4]   2> 438843 T811 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 438853 T856 C509 P49410 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/nodes_used_collection_shard2_replica1/data/index.20130522053949434
[junit4:junit4]   2> 438855 T856 C509 P49410 oasc.RecoveryStrategy.replay No replay needed. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 438855 T856 C509 P49410 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 438856 T856 C509 P49410 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=active
[junit4:junit4]   2> ASYNC  NEW_CORE C510 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore@3e5d57b0 url=http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2 node=127.0.0.1:49420_n_agm C510_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=active, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:49420_n_agm, base_url=http://127.0.0.1:49420/n_agm, leader=true}
[junit4:junit4]   2> 438855 T821 C510 P49420 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 438855 T811 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 438867 T856 C509 P49410 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=nodes_used_collection_shard2_replica1
[junit4:junit4]   2> 438957 T811 oass.IndexSchema.readSchema [awholynewcollection_0_shard4_replica4] Schema name=test
[junit4:junit4]   2> ASYNC  NEW_CORE C511 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore@3e5d57b0 url=http://127.0.0.1:49420/n_agm/nodes_used_collection_shard1_replica2 node=127.0.0.1:49420_n_agm C511_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{shard=shard1, state=active, core=nodes_used_collection_shard1_replica2, collection=nodes_used_collection, node_name=127.0.0.1:49420_n_agm, base_url=http://127.0.0.1:49420/n_agm, leader=true}
[junit4:junit4]   2> 438967 T821 C511 P49420 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@116023a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7bf47635),segFN=segments_1,generation=1}
[junit4:junit4]   2> 438968 T821 C511 P49420 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 439000 T821 C511 P49420 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@116023a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7bf47635),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@116023a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7bf47635),segFN=segments_2,generation=2}
[junit4:junit4]   2> 439001 T821 C511 P49420 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 439002 T821 C511 P49420 oass.SolrIndexSearcher.<init> Opening Searcher@6eb5fa93 realtime
[junit4:junit4]   2> 439002 T821 C511 P49420 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 439003 T821 C511 P49420 oasup.LogUpdateProcessor.finish [nodes_used_collection_shard1_replica2] webapp=/n_agm path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 148
[junit4:junit4]   2> ASYNC  NEW_CORE C512 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore@14eb7c05 url=http://127.0.0.1:49423/n_agm/nodes_used_collection_shard1_replica1 node=127.0.0.1:49423_n_agm C512_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49423_n_agm, base_url=http://127.0.0.1:49423/n_agm}
[junit4:junit4]   2> 439006 T857 C512 P49423 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 439006 T857 C512 P49423 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 439023 T822 C511 P49420 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 439024 T822 C511 P49420 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/n_agm path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=13 
[junit4:junit4]   2> 439026 T857 C512 P49423 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 439027 T857 C512 P49423 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 439027 T857 C512 P49423 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 439035 T826 C511 P49420 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/n_agm path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=4 
[junit4:junit4]   2> 439037 T857 C512 P49423 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 439049 T857 C512 P49423 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/index.20130522053949885
[junit4:junit4]   2> 439049 T857 C512 P49423 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@695f2c2e lockFactory=org.apache.lucene.store.NativeFSLockFactory@67dbf07c) fullCopy=false
[junit4:junit4]   2> 439070 T821 C511 P49420 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp=/n_agm path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> ASYNC  NEW_CORE C513 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore@14eb7c05 url=http://127.0.0.1:49423/n_agm/nodes_used_collection_shard1_replica1 node=127.0.0.1:49423_n_agm C513_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49423_n_agm, base_url=http://127.0.0.1:49423/n_agm}
[junit4:junit4]   2> 439175 T857 C513 P49423 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> ASYNC  NEW_CORE C514 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore@14eb7c05 url=http://127.0.0.1:49423/n_agm/nodes_used_collection_shard1_replica1 node=127.0.0.1:49423_n_agm C514_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=recovering, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:49423_n_agm, base_url=http://127.0.0.1:49423/n_agm}
[junit4:junit4]   2> 439229 T857 C514 P49423 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 439230 T857 C514 P49423 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 439233 T857 C514 P49423 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30b36590 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f0a457b),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30b36590 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f0a457b),segFN=segments_2,generation=2}
[junit4:junit4]   2> 439234 T857 C514 P49423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 439234 T857 C514 P49423 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 439235 T857 C514 P49423 oass.SolrIndexSearcher.<init> Opening Searcher@4cf9f95e main
[junit4:junit4]   2> 439237 T851 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher@4cf9f95e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 439303 T857 C514 P49423 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/index.20130522053949885 [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/index.20130522053949885;done=true>>]
[junit4:junit4]   2> 439322 T857 C514 P49423 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/index.20130522053949885
[junit4:junit4]   2> 439333 T857 C514 P49423 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/nodes_used_collection_shard1_replica1/data/index.20130522053949885
[junit4:junit4]   2> 439333 T857 C514 P49423 oasc.RecoveryStrategy.replay No replay needed. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 439334 T857 C514 P49423 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 439334 T857 C514 P49423 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=active
[junit4:junit4]   2> 439337 T857 C514 P49423 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=nodes_used_collection_shard1_replica1
[junit4:junit4]   2> 439941 T785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 439944 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:49410_n_agm_nodes_used_collection_shard2_replica1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49410_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49410/n_agm"}
[junit4:junit4]   2> 439962 T785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:49423_n_agm_nodes_used_collection_shard1_replica1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4]   2> 	  "collection":"nodes_used_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49423_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49423/n_agm"}
[junit4:junit4]   2> 440001 T816 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 440002 T861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 440003 T784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 440005 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 440006 T844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 440006 T802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 440004 T859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 440003 T789 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 448057 T793 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 448077 T793 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 448268 T793 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 448638 T793 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 448646 T793 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 448679 T812 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 448741 T793 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 448748 T793 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 448749 T793 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 448749 T793 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 448790 T793 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 448791 T793 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 448792 T793 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 448793 T793 oasc.SolrCore.<init> [awholynewcollection_0_shard4_replica1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard4_replica1/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard4_replica1/data/
[junit4:junit4]   2> 448793 T793 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15858f02
[junit4:junit4]   2> 448794 T793 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 448813 T793 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard4_replica1/data
[junit4:junit4]   2> 448833 T793 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard4_replica1/data/index/
[junit4:junit4]   2> 448834 T793 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard4_replica1] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard4_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 448833 T812 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 448836 T793 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard4_replica1/data/index
[junit4:junit4]   2> 448842 T812 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 448909 T793 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@554a1d7a lockFactory=org.apache.lucene.store.NativeFSLockFactory@2a48eda8),segFN=segments_1,generation=1}
[junit4:junit4]   2> 448910 T793 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 449047 T793 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 449048 T793 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 449049 T793 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 449051 T793 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 449052 T793 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 449053 T793 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 449053 T793 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 449061 T793 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 449062 T793 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 449156 T793 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 449223 T812 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 449245 T793 oass.SolrIndexSearcher.<init> Opening Searcher@41b6731b main
[junit4:junit4]   2> 449246 T793 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369215567659/awholynewcollection_0_shard4_replica1/data/tlog
[junit4:junit4]   2> 449333 T793 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 449334 T793 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 449343 T878 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard4_replica1] Registered new searcher Searcher@41b6731b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 449351 T793 oasc.ZkController.publish publishing core=awholynewcollection_0_shard4_replica1 state=down
[junit4:junit4]   2> 449379 T812 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 449493 T777 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 449628 T777 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 449634 T777 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 449668 T838 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 449754 T838 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 449807 T838 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 449811 T836 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 449818 T797 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 449834 T812 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 449838 T776 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 449852 T812 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 449855 T812 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 449855 T812 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 449884 T812 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 449885 T812 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 449886 T812 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 449886 T812 oasc.SolrCore.<init> [awholynewcollection_0_shard1_replica1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard1_replica1/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard1_replica1/data/
[junit4:junit4]   2> 449887 T812 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15858f02
[junit4:junit4]   2> 449888 T812 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 449906 T812 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard1_replica1/data
[junit4:junit4]   2> 449918 T812 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard1_replica1/data/index/
[junit4:junit4]   2> 449919 T812 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard1_replica1] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 449933 T797 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 449960 T812 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard1_replica1/data/index
[junit4:junit4]   2> 449990 T777 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 450000 T797 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 450015 T777 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 450020 T776 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 450066 T836 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 450072 T836 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 450111 T776 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 450121 T777 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 450131 T812 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4c0eeb1a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6dbc211e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 450132 T812 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 450153 T838 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 450162 T777 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 450163 T777 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 450164 T777 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 450229 T812 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 450230 T812 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 450231 T812 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 450233 T812 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 450234 T812 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 450234 T812 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 450235 T812 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 450236 T812 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 450237 T812 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 450257 T777 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 450262 T777 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 450263 T777 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 450264 T777 oasc.SolrCore.<init> [awholynewcollection_0_shard4_replica2] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard4_replica2/, dataDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard4_replica2/data/
[junit4:junit4]   2> 450265 T777 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15858f02
[junit4:junit4]   2> 450266 T777 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 450329 T777 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard4_replica2/data
[junit4:junit4]   2> 450350 T777 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard4_replica2/data/index/
[junit4:junit4]   2> 450350 T777 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard4_replica2] Solr index directory '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard4_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 450363 T777 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard4_replica2/data/index
[junit4:junit4]   2> 450374 T777 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5eb501ee lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d6c32dd),segFN=segments_1,generation=1}
[junit4:junit4]   2> 450375 T777 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 450383 T812 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 450391 T777 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 450392 T777 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 450397 T777 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 450399 T777 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 450448 T824 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 450452 T777 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 450452 T777 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 450453 T777 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 450454 T777 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 450455 T777 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 450494 T812 oass.SolrIndexSearcher.<init> Opening Searcher@2685334e main
[junit4:junit4]   2> 450495 T812 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369215569899/awholynewcollection_0_shard1_replica1/data/tlog
[junit4:junit4]   2> 450517 T812 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 450518 T812 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 450525 T879 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard1_replica1] Registered new searcher Searcher@2685334e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 450529 T797 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 450564 T836 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 450573 T838 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 450590 T812 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica1 state=down
[junit4:junit4]   2> 450624 T807 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 450631 T797 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 450636 T836 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 450671 T824 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 450703 T839 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 450724 T795 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 450759 T807 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 450751 T824 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 450743 T777 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 450809 T807 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 450825 T838 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 450833 T795 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 450842 T795 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 450875 T839 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 450883 T839 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 450910 T777 oass.SolrIndexSearcher.<init> Opening Searcher@61f31fff main
[junit4:junit4]   2> 450911 T777 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369215566388/awholynewcollection_0_shard4_replica2/data/tlog
[junit4:junit4]   2> 450939 T823 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 450945 T777 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 450946 T777 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 450951 T880 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard4_replica2] Registered new searcher Searcher@61f31fff main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 450957 T777 oasc.ZkController.publish publishing core=awholynewcollection_0_shard4_replica2 state=down
[junit4:junit4]   2> 450975 T823 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 451004 T823 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 451026 T780 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 451056 T811 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 451056 T780 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 451081 T780 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 451130 T811 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 451135 T811 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 451137 T825 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 451140 T808 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 451158 T825 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 451164 T825 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 451170 T808 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 451183 T808 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 452103 T766 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 1015ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4:junit4]   2> 452114 T836 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 452115 T838 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<i

[...truncated too long message...]

ata/index
[junit4:junit4]   2> 506795 T762 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_1_shard4_replica2/data [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_1_shard4_replica2/data;done=false>>]
[junit4:junit4]   2> 506795 T762 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_1_shard4_replica2/data
[junit4:junit4]   2> 506805 T762 oasc.SolrCore.close [awholynewcollection_1_shard2_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@70d90a43
[junit4:junit4]   2> 506819 T762 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 506820 T762 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 506824 T762 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 506824 T762 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 506827 T762 oasc.SolrCore.closeSearcher [awholynewcollection_1_shard2_replica2] Closing main searcher on request.
[junit4:junit4]   2> 506828 T762 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4]   2> 506828 T762 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_1_shard2_replica2/data/index [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_1_shard2_replica2/data/index;done=false>>]
[junit4:junit4]   2> 506829 T762 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_1_shard2_replica2/data/index
[junit4:junit4]   2> 506829 T762 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_1_shard2_replica2/data [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_1_shard2_replica2/data;done=false>>]
[junit4:junit4]   2> 506830 T762 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369215574745/awholynewcollection_1_shard2_replica2/data
[junit4:junit4]   2> 506827 T1086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:49423_n_agm_awholynewcollection_0_shard2_replica3",
[junit4:junit4]   2> 	  "numShards":"4",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"awholynewcollection_0_shard2_replica3",
[junit4:junit4]   2> 	  "collection":"awholynewcollection_0",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49423_n_agm",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49423/n_agm"}
[junit4:junit4]   2> 506863 T762 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/n_agm,null}
[junit4:junit4]   2> 506926 T762 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 506937 T762 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49407 49407
[junit4:junit4]   1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=12BA1200442C22A9 -Dtests.slow=true -Dtests.locale=sr_CS -Dtests.timezone=America/Asuncion -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR   92.2s | CollectionsAPIDistributedZkTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: collection already exists: awholynewcollection_1
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([12BA1200442C22A9:935C9C1833734295]:0)
[junit4:junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:424)
[junit4:junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:264)
[junit4:junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:306)
[junit4:junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1400)
[junit4:junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:438)
[junit4:junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:146)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 507272 T762 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 92218 T761 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4]   2> 508140 T844 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 508141 T844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 508141 T844 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=65), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=65)), sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=sr_CS, timezone=America/Asuncion
[junit4:junit4]   2> NOTE: Mac OS X 10.8.3 x86_64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=2,threads=2,free=59159336,total=169865216
[junit4:junit4]   2> NOTE: All tests run in this JVM: [StandardRequestHandlerTest, TestSurroundQueryParser, FileUtilsTest, URLClassifyProcessorTest, ClusterStateTest, ConvertedLegacyTest, DateMathParserTest, TestExtendedDismaxParser, SolrPluginUtilsTest, TestPropInject, PreAnalyzedFieldTest, SolrCmdDistributorTest, DisMaxRequestHandlerTest, DefaultValueUpdateProcessorTest, SimpleFacetsTest, TestMergePolicyConfig, TestPseudoReturnFields, HighlighterConfigTest, TestFieldCollectionResource, SyncSliceTest, TestMaxScoreQueryParser, RequiredFieldsTest, TestQuerySenderNoQuery, ShardRoutingCustomTest, TestQueryTypes, TestSchemaResource, DOMUtilTest, TestFieldResource, TestAddFieldRealTimeGet, TestCloudManagedSchemaAddField, CacheHeaderTest, IndexReaderFactoryTest, NumericFieldsTest, TestJmxIntegration, TestSolrJ, TestCoreContainer, MultiTermTest, SolrIndexSplitterTest, TestPerFieldSimilarity, CollectionsAPIDistributedZkTest]
[junit4:junit4] Completed in 93.31s, 1 test, 1 error <<< FAILURES!

[...truncated 789 lines...]
BUILD FAILED
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:380: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:360: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:181: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:437: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1240: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:884: There were test failures: 295 suites, 1231 tests, 1 error, 18 ignored (6 assumptions)

Total time: 104 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 492 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/492/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch

Error Message:
Server at http://127.0.0.1:50341/m_o returned non ok status:500, message:Server Error

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:50341/m_o returned non ok status:500, message:Server Error
	at __randomizedtesting.SeedInfo.seed([A7CD213BFA1FDE57:262BAF238D40BE6B]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
	at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
	at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 9482 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4]   2> 2208047 T3585 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /m_o/
[junit4:junit4]   2> 2208057 T3585 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyShardSplitTest-1369193690262
[junit4:junit4]   2> 2208063 T3585 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 2208068 T3586 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 2208168 T3585 oasc.ZkTestServer.run start zk server on port:50334
[junit4:junit4]   2> 2208174 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2208184 T3592 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73ab2f1d name:ZooKeeperConnection Watcher:127.0.0.1:50334 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2208184 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2208185 T3585 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 2208208 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2208216 T3594 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66a91f35 name:ZooKeeperConnection Watcher:127.0.0.1:50334/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2208217 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2208218 T3585 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 2208235 T3585 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 2208244 T3585 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 2208253 T3585 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 2208264 T3585 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2208269 T3585 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2208291 T3585 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 2208292 T3585 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 2208302 T3585 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 2208303 T3585 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 2208313 T3585 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 2208315 T3585 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 2208324 T3585 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 2208325 T3585 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 2208342 T3585 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 2208344 T3585 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 2208357 T3585 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 2208359 T3585 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 2208377 T3585 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 2208379 T3585 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 2208395 T3585 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 2208402 T3585 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 2209105 T3585 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2209117 T3585 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50337
[junit4:junit4]   2> 2209120 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2209123 T3585 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2209124 T3585 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369193690624
[junit4:junit4]   2> 2209128 T3585 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369193690624/solr.xml
[junit4:junit4]   2> 2209129 T3585 oasc.CoreContainer.<init> New CoreContainer 1921044184
[junit4:junit4]   2> 2209129 T3585 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369193690624/'
[junit4:junit4]   2> 2209130 T3585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369193690624/'
[junit4:junit4]   2> 2209362 T3585 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2209363 T3585 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2209364 T3585 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2209365 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2209366 T3585 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2209367 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2209368 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2209369 T3585 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2209370 T3585 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2209371 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2209413 T3585 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2209416 T3585 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50334/solr
[junit4:junit4]   2> 2209423 T3585 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2209426 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2209433 T3605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@639d0708 name:ZooKeeperConnection Watcher:127.0.0.1:50334 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2209434 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2209442 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2209468 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2209474 T3607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6075ace4 name:ZooKeeperConnection Watcher:127.0.0.1:50334/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2209474 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2209481 T3585 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 2209494 T3585 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 2209504 T3585 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 2209510 T3585 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50337_m_o
[junit4:junit4]   2> 2209514 T3585 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50337_m_o
[junit4:junit4]   2> 2209532 T3585 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 2209551 T3585 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 2209558 T3585 oasc.Overseer.start Overseer (id=89731477686779907-127.0.0.1:50337_m_o-n_0000000000) starting
[junit4:junit4]   2> 2209569 T3585 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 2209591 T3609 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 2209593 T3585 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 2209604 T3585 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 2209611 T3585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2209620 T3608 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 2209633 T3610 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369193690624/collection1
[junit4:junit4]   2> 2209634 T3610 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 2209635 T3610 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2209636 T3610 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 2209640 T3610 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369193690624/collection1/'
[junit4:junit4]   2> 2209642 T3610 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369193690624/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2209643 T3610 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369193690624/collection1/lib/README' to classloader
[junit4:junit4]   2> 2209750 T3610 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2209893 T3610 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2209897 T3610 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2209905 T3610 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 2211036 T3610 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2211042 T3610 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2211045 T3610 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2211055 T3610 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2211137 T3610 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2211138 T3610 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369193690624/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/control/data/
[junit4:junit4]   2> 2211138 T3610 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59e50a34
[junit4:junit4]   2> 2211139 T3610 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2211140 T3610 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/control/data
[junit4:junit4]   2> 2211140 T3610 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/control/data/index/
[junit4:junit4]   2> 2211141 T3610 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2211153 T3610 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/control/data/index
[junit4:junit4]   2> 2211206 T3610 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3f032104 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d4e70b7),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2211207 T3610 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2211212 T3610 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2211213 T3610 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2211214 T3610 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2211215 T3610 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2211216 T3610 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2211219 T3610 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2211220 T3610 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2211221 T3610 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2211221 T3610 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2211257 T3610 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2211271 T3610 oass.SolrIndexSearcher.<init> Opening Searcher@5df36f2 main
[junit4:junit4]   2> 2211272 T3610 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/control/data/tlog
[junit4:junit4]   2> 2211273 T3610 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2211274 T3610 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2211278 T3611 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5df36f2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2211280 T3610 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2211281 T3610 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2212642 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2212644 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50337_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50337/m_o"}
[junit4:junit4]   2> 2212645 T3608 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2212645 T3608 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 2212658 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2213293 T3610 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2213294 T3610 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50337/m_o collection:control_collection shard:shard1
[junit4:junit4]   2> 2213295 T3610 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2213317 T3610 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2213324 T3610 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2213324 T3610 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2213325 T3610 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50337/m_o/collection1/
[junit4:junit4]   2> 2213325 T3610 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2213325 T3610 oasc.SyncStrategy.syncToMe http://127.0.0.1:50337/m_o/collection1/ has no replicas
[junit4:junit4]   2> 2213326 T3610 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50337/m_o/collection1/
[junit4:junit4]   2> 2213326 T3610 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2214167 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2214190 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2214231 T3610 oasc.ZkController.register We are http://127.0.0.1:50337/m_o/collection1/ and leader is http://127.0.0.1:50337/m_o/collection1/
[junit4:junit4]   2> 2214232 T3610 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50337/m_o
[junit4:junit4]   2> 2214233 T3610 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2214233 T3610 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2214234 T3610 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2214244 T3610 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2214246 T3585 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2214246 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2214247 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2214286 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 2214295 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2214296 T3614 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16ba2ee0 name:ZooKeeperConnection Watcher:127.0.0.1:50334/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2214299 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2214304 T3585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2214311 T3585 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 2214911 T3585 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2214919 T3585 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50341
[junit4:junit4]   2> 2214921 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2214922 T3585 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2214922 T3585 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369193696517
[junit4:junit4]   2> 2214923 T3585 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369193696517/solr.xml
[junit4:junit4]   2> 2214923 T3585 oasc.CoreContainer.<init> New CoreContainer 289891141
[junit4:junit4]   2> 2214924 T3585 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369193696517/'
[junit4:junit4]   2> 2214925 T3585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369193696517/'
[junit4:junit4]   2> 2215131 T3585 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2215135 T3585 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2215136 T3585 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2215137 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2215137 T3585 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2215138 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2215139 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2215139 T3585 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2215141 T3585 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2215141 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2215172 T3585 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2215173 T3585 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50334/solr
[junit4:junit4]   2> 2215173 T3585 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2215178 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2215181 T3625 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ed78655 name:ZooKeeperConnection Watcher:127.0.0.1:50334 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2215183 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2215188 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2215222 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2215228 T3627 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ba9e08d name:ZooKeeperConnection Watcher:127.0.0.1:50334/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2215228 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2215242 T3585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2215722 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2215724 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50337_m_o_collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50337_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50337/m_o"}
[junit4:junit4]   2> 2215741 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2215742 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2215741 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2216253 T3585 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50341_m_o
[junit4:junit4]   2> 2216297 T3585 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50341_m_o
[junit4:junit4]   2> 2216314 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2216315 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2216317 T3607 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2216317 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2216328 T3627 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2216329 T3614 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2216347 T3628 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369193696517/collection1
[junit4:junit4]   2> 2216348 T3628 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 2216350 T3628 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2216350 T3628 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 2216357 T3628 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369193696517/collection1/'
[junit4:junit4]   2> 2216360 T3628 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369193696517/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2216361 T3628 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369193696517/collection1/lib/README' to classloader
[junit4:junit4]   2> 2216456 T3628 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2216567 T3628 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2216571 T3628 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2216578 T3628 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 2217513 T3628 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2217517 T3628 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2217520 T3628 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2217535 T3628 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2217714 T3628 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2217714 T3628 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369193696517/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty1/
[junit4:junit4]   2> 2217715 T3628 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59e50a34
[junit4:junit4]   2> 2217716 T3628 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2217716 T3628 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty1
[junit4:junit4]   2> 2217717 T3628 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty1/index/
[junit4:junit4]   2> 2217717 T3628 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2217719 T3628 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty1/index
[junit4:junit4]   2> 2217724 T3628 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@10a19980 lockFactory=org.apache.lucene.store.NativeFSLockFactory@732e2e1e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2217725 T3628 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2217731 T3628 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2217732 T3628 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2217733 T3628 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2217734 T3628 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2217734 T3628 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2217735 T3628 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2217735 T3628 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2217740 T3628 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2217743 T3628 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2217785 T3628 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2217804 T3628 oass.SolrIndexSearcher.<init> Opening Searcher@2fe87465 main
[junit4:junit4]   2> 2217805 T3628 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty1/tlog
[junit4:junit4]   2> 2217806 T3628 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2217807 T3628 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2217819 T3629 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2fe87465 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2217823 T3628 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2217824 T3628 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2218770 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2218773 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50341_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50341/m_o"}
[junit4:junit4]   2> 2218773 T3608 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 2218774 T3608 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 2218789 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2218789 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2218800 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2218840 T3628 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2218841 T3628 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50341/m_o collection:collection1 shard:shard1
[junit4:junit4]   2> 2218843 T3628 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2218868 T3628 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2218877 T3628 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2218877 T3628 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2218878 T3628 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50341/m_o/collection1/
[junit4:junit4]   2> 2218878 T3628 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2218879 T3628 oasc.SyncStrategy.syncToMe http://127.0.0.1:50341/m_o/collection1/ has no replicas
[junit4:junit4]   2> 2218879 T3628 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50341/m_o/collection1/
[junit4:junit4]   2> 2218879 T3628 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 2220312 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2220350 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2220350 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2220351 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2220389 T3628 oasc.ZkController.register We are http://127.0.0.1:50341/m_o/collection1/ and leader is http://127.0.0.1:50341/m_o/collection1/
[junit4:junit4]   2> 2220390 T3628 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50341/m_o
[junit4:junit4]   2> 2220390 T3628 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2220390 T3628 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2220391 T3628 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2220395 T3628 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2220398 T3585 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2220399 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2220399 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2220967 T3585 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2220979 T3585 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50344
[junit4:junit4]   2> 2220980 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2220980 T3585 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2220981 T3585 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369193702636
[junit4:junit4]   2> 2220981 T3585 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369193702636/solr.xml
[junit4:junit4]   2> 2220982 T3585 oasc.CoreContainer.<init> New CoreContainer 1427120422
[junit4:junit4]   2> 2220983 T3585 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369193702636/'
[junit4:junit4]   2> 2220984 T3585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369193702636/'
[junit4:junit4]   2> 2221226 T3585 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2221226 T3585 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2221227 T3585 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2221228 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2221228 T3585 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2221229 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2221229 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2221229 T3585 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2221230 T3585 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2221231 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2221257 T3585 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2221260 T3585 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50334/solr
[junit4:junit4]   2> 2221260 T3585 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2221264 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2221273 T3641 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@595f9 name:ZooKeeperConnection Watcher:127.0.0.1:50334 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2221275 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2221283 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2221307 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2221312 T3643 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75469a41 name:ZooKeeperConnection Watcher:127.0.0.1:50334/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2221312 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2221326 T3585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2221918 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2221920 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50341_m_o_collection1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50341_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50341/m_o"}
[junit4:junit4]   2> 2221961 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2221962 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2221962 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2221961 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2222339 T3585 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50344_m_o
[junit4:junit4]   2> 2222342 T3585 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50344_m_o
[junit4:junit4]   2> 2222348 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2222348 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 2222351 T3607 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2222352 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2222351 T3643 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2222352 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2222359 T3614 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2222366 T3627 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 2222374 T3644 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369193702636/collection1
[junit4:junit4]   2> 2222374 T3644 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 2222376 T3644 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2222376 T3644 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 2222380 T3644 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369193702636/collection1/'
[junit4:junit4]   2> 2222381 T3644 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369193702636/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2222382 T3644 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369193702636/collection1/lib/README' to classloader
[junit4:junit4]   2> 2222472 T3644 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2222564 T3644 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2222568 T3644 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2222576 T3644 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 2223513 T3644 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2223526 T3644 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2223529 T3644 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2223536 T3644 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2223616 T3644 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2223617 T3644 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369193702636/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty2/
[junit4:junit4]   2> 2223617 T3644 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59e50a34
[junit4:junit4]   2> 2223618 T3644 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2223619 T3644 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty2
[junit4:junit4]   2> 2223620 T3644 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty2/index/
[junit4:junit4]   2> 2223620 T3644 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2223621 T3644 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty2/index
[junit4:junit4]   2> 2223626 T3644 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e77aeee lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c957739),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2223627 T3644 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2223631 T3644 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2223631 T3644 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2223632 T3644 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2223639 T3644 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2223641 T3644 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2223641 T3644 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2223642 T3644 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2223643 T3644 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2223644 T3644 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2223664 T3644 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2223674 T3644 oass.SolrIndexSearcher.<init> Opening Searcher@5f8d8b15 main
[junit4:junit4]   2> 2223675 T3644 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty2/tlog
[junit4:junit4]   2> 2223677 T3644 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2223678 T3644 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2223684 T3645 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f8d8b15 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2223687 T3644 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2223688 T3644 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2225012 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2225014 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50344_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50344/m_o"}
[junit4:junit4]   2> 2225015 T3608 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 2225015 T3608 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4]   2> 2225028 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225028 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225030 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225031 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2225694 T3644 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2225695 T3644 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50344/m_o collection:collection1 shard:shard2
[junit4:junit4]   2> 2225697 T3644 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 2225720 T3644 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2225727 T3644 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2225727 T3644 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2225727 T3644 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50344/m_o/collection1/
[junit4:junit4]   2> 2225728 T3644 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2225728 T3644 oasc.SyncStrategy.syncToMe http://127.0.0.1:50344/m_o/collection1/ has no replicas
[junit4:junit4]   2> 2225728 T3644 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50344/m_o/collection1/
[junit4:junit4]   2> 2225728 T3644 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 2226552 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2226577 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2226577 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2226578 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2226578 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2226626 T3644 oasc.ZkController.register We are http://127.0.0.1:50344/m_o/collection1/ and leader is http://127.0.0.1:50344/m_o/collection1/
[junit4:junit4]   2> 2226626 T3644 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50344/m_o
[junit4:junit4]   2> 2226627 T3644 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2226627 T3644 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2226627 T3644 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2226632 T3644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2226641 T3585 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2226642 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2226642 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2227290 T3585 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2227298 T3585 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50347
[junit4:junit4]   2> 2227300 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2227300 T3585 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2227301 T3585 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369193708875
[junit4:junit4]   2> 2227301 T3585 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369193708875/solr.xml
[junit4:junit4]   2> 2227302 T3585 oasc.CoreContainer.<init> New CoreContainer 976310575
[junit4:junit4]   2> 2227303 T3585 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369193708875/'
[junit4:junit4]   2> 2227303 T3585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369193708875/'
[junit4:junit4]   2> 2227553 T3585 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2227554 T3585 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2227555 T3585 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2227555 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2227558 T3585 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2227558 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2227559 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2227560 T3585 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2227560 T3585 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2227561 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2227587 T3585 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2227588 T3585 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50334/solr
[junit4:junit4]   2> 2227589 T3585 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2227592 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2227603 T3657 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15981cd8 name:ZooKeeperConnection Watcher:127.0.0.1:50334 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2227604 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2227611 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2227642 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2227652 T3659 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cd55d07 name:ZooKeeperConnection Watcher:127.0.0.1:50334/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2227653 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2227687 T3585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2228099 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2228101 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50344_m_o_collection1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50344_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50344/m_o"}
[junit4:junit4]   2> 2228118 T3659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2228118 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2228119 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2228119 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2228120 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2228699 T3585 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50347_m_o
[junit4:junit4]   2> 2228702 T3585 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50347_m_o
[junit4:junit4]   2> 2228710 T3659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2228711 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2228711 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 2228718 T3607 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2228718 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2228720 T3643 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2228720 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2228739 T3659 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2228740 T3627 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2228739 T3614 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 2228754 T3660 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369193708875/collection1
[junit4:junit4]   2> 2228754 T3660 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 2228757 T3660 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2228757 T3660 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 2228761 T3660 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369193708875/collection1/'
[junit4:junit4]   2> 2228763 T3660 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369193708875/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2228764 T3660 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369193708875/collection1/lib/README' to classloader
[junit4:junit4]   2> 2228851 T3660 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2228955 T3660 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2228959 T3660 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2228965 T3660 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 2229822 T3660 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2229827 T3660 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2229835 T3660 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2229848 T3660 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2229924 T3660 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2229925 T3660 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369193708875/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/
[junit4:junit4]   2> 2229925 T3660 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59e50a34
[junit4:junit4]   2> 2229926 T3660 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2229927 T3660 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3
[junit4:junit4]   2> 2229927 T3660 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/index/
[junit4:junit4]   2> 2229927 T3660 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2229929 T3660 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/index
[junit4:junit4]   2> 2229934 T3660 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6baff476 lockFactory=org.apache.lucene.store.NativeFSLockFactory@521dfe59),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2229935 T3660 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2229940 T3660 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2229940 T3660 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2229942 T3660 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2229943 T3660 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2229944 T3660 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2229944 T3660 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2229945 T3660 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2229946 T3660 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2229947 T3660 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2229973 T3660 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2229983 T3660 oass.SolrIndexSearcher.<init> Opening Searcher@531890e3 main
[junit4:junit4]   2> 2229984 T3660 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/tlog
[junit4:junit4]   2> 2229985 T3660 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2229986 T3660 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2229992 T3661 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@531890e3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2229996 T3660 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2229996 T3660 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2231151 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2231153 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50347_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50347/m_o"}
[junit4:junit4]   2> 2231154 T3608 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 2231154 T3608 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 2231170 T3659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2231170 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2231170 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2231170 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2231170 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2232002 T3660 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2232003 T3660 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50347/m_o collection:collection1 shard:shard1
[junit4:junit4]   2> 2232012 T3660 oasc.ZkController.register We are http://127.0.0.1:50347/m_o/collection1/ and leader is http://127.0.0.1:50341/m_o/collection1/
[junit4:junit4]   2> 2232013 T3660 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50347/m_o
[junit4:junit4]   2> 2232013 T3660 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 2232014 T3660 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C1287 name=collection1 org.apache.solr.core.SolrCore@46ed285f url=http://127.0.0.1:50347/m_o/collection1 node=127.0.0.1:50347_m_o C1287_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:50347_m_o, base_url=http://127.0.0.1:50347/m_o}
[junit4:junit4]   2> 2232016 T3662 C1287 P50347 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 2232016 T3660 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2232016 T3662 C1287 P50347 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 2232017 T3662 C1287 P50347 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 2232017 T3662 C1287 P50347 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2232021 T3585 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2232022 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2232023 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2232029 T3662 C1287 P50347 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2232064 T3618 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:50347_m_o_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 2232612 T3585 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 2232621 T3585 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50351
[junit4:junit4]   2> 2232623 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2232623 T3585 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2232624 T3585 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369193714254
[junit4:junit4]   2> 2232625 T3585 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369193714254/solr.xml
[junit4:junit4]   2> 2232625 T3585 oasc.CoreContainer.<init> New CoreContainer 1527064045
[junit4:junit4]   2> 2232626 T3585 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369193714254/'
[junit4:junit4]   2> 2232627 T3585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369193714254/'
[junit4:junit4]   2> 2232686 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2232688 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50347_m_o_collection1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50347_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50347/m_o"}
[junit4:junit4]   2> 2232709 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2232710 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2232710 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2232710 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2232710 T3659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2232883 T3585 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 2232884 T3585 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2232885 T3585 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 2232885 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2232886 T3585 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2232886 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2232888 T3585 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2232888 T3585 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2232889 T3585 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2232890 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 2232918 T3585 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 2232919 T3585 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50334/solr
[junit4:junit4]   2> 2232920 T3585 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 2232925 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2232928 T3674 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2099a9b6 name:ZooKeeperConnection Watcher:127.0.0.1:50334 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2232931 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2232940 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 2232967 T3585 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2232971 T3676 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2dbf0ba1 name:ZooKeeperConnection Watcher:127.0.0.1:50334/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2232971 T3585 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2232984 T3585 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2233067 T3618 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:50347_m_o_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 2233068 T3618 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:50347_m_o_collection1&state=recovering&nodeName=127.0.0.1:50347_m_o&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1004 
[junit4:junit4]   2> 2233996 T3585 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50351_m_o
[junit4:junit4]   2> 2233999 T3585 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50351_m_o
[junit4:junit4]   2> 2234010 T3659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2234010 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2234010 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 2234018 T3643 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2234018 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2234045 T3659 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2234047 T3614 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2234047 T3607 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2234048 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2234048 T3676 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2234048 T3627 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 2234068 T3677 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369193714254/collection1
[junit4:junit4]   2> 2234068 T3677 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 2234070 T3677 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2234070 T3677 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 2234072 T3677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369193714254/collection1/'
[junit4:junit4]   2> 2234077 T3677 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369193714254/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2234078 T3677 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369193714254/collection1/lib/README' to classloader
[junit4:junit4]   2> 2234181 T3677 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2234308 T3677 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2234312 T3677 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 2234323 T3677 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> ASYNC  NEW_CORE C1288 name=collection1 org.apache.solr.core.SolrCore@46ed285f url=http://127.0.0.1:50347/m_o/collection1 node=127.0.0.1:50347_m_o C1288_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:50347_m_o, base_url=http://127.0.0.1:50347/m_o}
[junit4:junit4]   2> 2235070 T3662 C1288 P50347 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50341/m_o/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 2235071 T3662 C1288 P50347 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50347/m_o START replicas=[http://127.0.0.1:50341/m_o/collection1/] nUpdates=100
[junit4:junit4]   2> 2235075 T3662 C1288 P50347 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 2235083 T3662 C1288 P50347 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 2235083 T3662 C1288 P50347 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 2235083 T3662 C1288 P50347 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 2235084 T3662 C1288 P50347 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 2235084 T3662 C1288 P50347 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50341/m_o/collection1/. core=collection1
[junit4:junit4]   2> 2235084 T3662 C1288 P50347 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C1289 name=collection1 org.apache.solr.core.SolrCore@58e9c868 url=http://127.0.0.1:50341/m_o/collection1 node=127.0.0.1:50341_m_o C1289_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50341_m_o, base_url=http://127.0.0.1:50341/m_o, leader=true}
[junit4:junit4]   2> 2235116 T3621 C1289 P50341 oasc.SolrCore.execute [collection1] webapp=/m_o path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 2235127 T3622 C1289 P50341 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2235149 T3622 C1289 P50341 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@10a19980 lockFactory=org.apache.lucene.store.NativeFSLockFactory@732e2e1e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2235153 T3622 C1289 P50341 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2235163 T3622 C1289 P50341 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@10a19980 lockFactory=org.apache.lucene.store.NativeFSLockFactory@732e2e1e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@10a19980 lockFactory=org.apache.lucene.store.NativeFSLockFactory@732e2e1e),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2235163 T3622 C1289 P50341 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2235164 T3622 C1289 P50341 oass.SolrIndexSearcher.<init> Opening Searcher@224f1d7e realtime
[junit4:junit4]   2> 2235164 T3622 C1289 P50341 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2235165 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 39
[junit4:junit4]   2> 2235171 T3662 C1288 P50347 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2235171 T3662 C1288 P50347 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 2235190 T3619 C1289 P50341 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2235192 T3619 C1289 P50341 oasc.SolrCore.execute [collection1] webapp=/m_o path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=8 
[junit4:junit4]   2> 2235197 T3662 C1288 P50347 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 2235197 T3662 C1288 P50347 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 2235197 T3662 C1288 P50347 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 2235207 T3620 C1289 P50341 oasc.SolrCore.execute [collection1] webapp=/m_o path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=4 
[junit4:junit4]   2> 2235208 T3662 C1288 P50347 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 2235211 T3662 C1288 P50347 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/index.20130521223517411
[junit4:junit4]   2> 2235211 T3662 C1288 P50347 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@1f893d0d lockFactory=org.apache.lucene.store.NativeFSLockFactory@c78aa9a) fullCopy=false
[junit4:junit4]   2> 2235214 T3618 C1289 P50341 oasc.SolrCore.execute [collection1] webapp=/m_o path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 2235216 T3662 C1288 P50347 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 2235219 T3662 C1288 P50347 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 2235219 T3662 C1288 P50347 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 2235222 T3662 C1288 P50347 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6baff476 lockFactory=org.apache.lucene.store.NativeFSLockFactory@521dfe59),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6baff476 lockFactory=org.apache.lucene.store.NativeFSLockFactory@521dfe59),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2235223 T3662 C1288 P50347 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2235224 T3662 C1288 P50347 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 2235224 T3662 C1288 P50347 oass.SolrIndexSearcher.<init> Opening Searcher@9a78115 main
[junit4:junit4]   2> 2235226 T3661 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9a78115 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2235227 T3662 C1288 P50347 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/index.20130521223517411 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/index.20130521223517411;done=true>>]
[junit4:junit4]   2> 2235227 T3662 C1288 P50347 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/index.20130521223517411
[junit4:junit4]   2> 2235227 T3662 C1288 P50347 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty3/index.20130521223517411
[junit4:junit4]   2> 2235228 T3662 C1288 P50347 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 2235228 T3662 C1288 P50347 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 2235228 T3662 C1288 P50347 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2235229 T3662 C1288 P50347 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2235231 T3662 C1288 P50347 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 2235291 T3677 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2235296 T3677 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2235299 T3677 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2235308 T3677 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2235384 T3677 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2235384 T3677 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369193714254/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/
[junit4:junit4]   2> 2235385 T3677 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59e50a34
[junit4:junit4]   2> 2235385 T3677 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2235386 T3677 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4
[junit4:junit4]   2> 2235387 T3677 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index/
[junit4:junit4]   2> 2235387 T3677 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2235388 T3677 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index
[junit4:junit4]   2> 2235393 T3677 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@630785c8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c0e8559),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2235393 T3677 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2235398 T3677 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2235398 T3677 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2235404 T3677 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2235407 T3677 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2235410 T3677 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2235411 T3677 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2235413 T3677 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2235414 T3677 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2235414 T3677 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2235435 T3677 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2235447 T3677 oass.SolrIndexSearcher.<init> Opening Searcher@503b362a main
[junit4:junit4]   2> 2235447 T3677 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/tlog
[junit4:junit4]   2> 2235449 T3677 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2235449 T3677 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2235456 T3680 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@503b362a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2235459 T3677 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2235460 T3677 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2235753 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2235755 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50347_m_o_collection1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50347_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50347/m_o"}
[junit4:junit4]   2> 2235766 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50351_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50351/m_o"}
[junit4:junit4]   2> 2235767 T3608 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 2235767 T3608 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4]   2> 2235781 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235782 T3659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235784 T3676 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235784 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235785 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2235790 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2236473 T3677 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2236474 T3677 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50351/m_o collection:collection1 shard:shard2
[junit4:junit4]   2> 2236482 T3677 oasc.ZkController.register We are http://127.0.0.1:50351/m_o/collection1/ and leader is http://127.0.0.1:50344/m_o/collection1/
[junit4:junit4]   2> 2236483 T3677 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50351/m_o
[junit4:junit4]   2> 2236483 T3677 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 2236484 T3677 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C1290 name=collection1 org.apache.solr.core.SolrCore@42c25f65 url=http://127.0.0.1:50351/m_o/collection1 node=127.0.0.1:50351_m_o C1290_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:50351_m_o, base_url=http://127.0.0.1:50351/m_o}
[junit4:junit4]   2> 2236485 T3681 C1290 P50351 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 2236485 T3681 C1290 P50351 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 2236485 T3681 C1290 P50351 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 2236486 T3681 C1290 P50351 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2236490 T3677 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2236492 T3681 C1290 P50351 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2236521 T3585 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2236523 T3585 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2236524 T3585 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2236551 T3585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2236554 T3635 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:50351_m_o_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 2236556 T3585 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 2236557 T3585 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 2236559 T3585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2237308 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2237310 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50351_m_o_collection1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50351_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50351/m_o"}
[junit4:junit4]   2> 2237321 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2237322 T3676 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2237323 T3659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2237324 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2237324 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2237324 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2237556 T3635 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:50351_m_o_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 2237556 T3635 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:50351_m_o_collection1&state=recovering&nodeName=127.0.0.1:50351_m_o&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002 
[junit4:junit4]   2> 2237565 T3585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2238578 T3585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2>  C1290_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:50351_m_o, base_url=http://127.0.0.1:50351/m_o}
[junit4:junit4]   2> 2239559 T3681 C1290 P50351 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50344/m_o/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 2239560 T3681 C1290 P50351 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50351/m_o START replicas=[http://127.0.0.1:50344/m_o/collection1/] nUpdates=100
[junit4:junit4]   2> 2239563 T3681 C1290 P50351 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 2239564 T3681 C1290 P50351 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 2239565 T3681 C1290 P50351 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 2239566 T3681 C1290 P50351 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 2239566 T3681 C1290 P50351 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 2239568 T3681 C1290 P50351 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50344/m_o/collection1/. core=collection1
[junit4:junit4]   2> 2239568 T3681 C1290 P50351 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C1291 name=collection1 org.apache.solr.core.SolrCore@7b87afc9 url=http://127.0.0.1:50344/m_o/collection1 node=127.0.0.1:50344_m_o C1291_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50344_m_o, base_url=http://127.0.0.1:50344/m_o, leader=true}
[junit4:junit4]   2> 2239585 T3635 C1291 P50344 oasc.SolrCore.execute [collection1] webapp=/m_o path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 2239588 T3585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2239594 T3639 C1291 P50344 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2239598 T3639 C1291 P50344 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e77aeee lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c957739),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2239599 T3639 C1291 P50344 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2239601 T3639 C1291 P50344 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e77aeee lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c957739),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e77aeee lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c957739),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2239601 T3639 C1291 P50344 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2239602 T3639 C1291 P50344 oass.SolrIndexSearcher.<init> Opening Searcher@144ca823 realtime
[junit4:junit4]   2> 2239603 T3639 C1291 P50344 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2239603 T3639 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 2239605 T3681 C1290 P50351 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2239605 T3681 C1290 P50351 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 2239611 T3636 C1291 P50344 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2239612 T3636 C1291 P50344 oasc.SolrCore.execute [collection1] webapp=/m_o path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 2239613 T3681 C1290 P50351 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 2239613 T3681 C1290 P50351 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 2239613 T3681 C1290 P50351 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 2239616 T3637 C1291 P50344 oasc.SolrCore.execute [collection1] webapp=/m_o path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 2239617 T3681 C1290 P50351 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 2239619 T3681 C1290 P50351 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index.20130521223521820
[junit4:junit4]   2> 2239620 T3681 C1290 P50351 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@d0ee16c lockFactory=org.apache.lucene.store.NativeFSLockFactory@3f238a85) fullCopy=false
[junit4:junit4]   2> 2239623 T3635 C1291 P50344 oasc.SolrCore.execute [collection1] webapp=/m_o path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1 
[junit4:junit4]   2> 2239630 T3681 C1290 P50351 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 2239634 T3681 C1290 P50351 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 2239634 T3681 C1290 P50351 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 2239636 T3681 C1290 P50351 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@630785c8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c0e8559),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@630785c8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c0e8559),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2239637 T3681 C1290 P50351 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2239638 T3681 C1290 P50351 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 2239638 T3681 C1290 P50351 oass.SolrIndexSearcher.<init> Opening Searcher@290d295 main
[junit4:junit4]   2> 2239640 T3680 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@290d295 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2239642 T3681 C1290 P50351 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index.20130521223521820 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index.20130521223521820;done=true>>]
[junit4:junit4]   2> 2239643 T3681 C1290 P50351 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index.20130521223521820
[junit4:junit4]   2> 2239643 T3681 C1290 P50351 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index.20130521223521820
[junit4:junit4]   2> 2239644 T3681 C1290 P50351 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 2239644 T3681 C1290 P50351 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 2239644 T3681 C1290 P50351 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 2239644 T3681 C1290 P50351 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2239647 T3681 C1290 P50351 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 2240353 T3608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2240354 T3608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:50351_m_o_collection1",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50351_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50351/m_o"}
[junit4:junit4]   2> 2240371 T3607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2240372 T3676 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2240372 T3643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2240372 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2240372 T3659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2240372 T3627 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 2240593 T3585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2240596 T3585 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C1292 name=collection1 org.apache.solr.core.SolrCore@2a8a13e5 url=http://127.0.0.1:50337/m_o/collection1 node=127.0.0.1:50337_m_o C1292_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:50337_m_o, base_url=http://127.0.0.1:50337/m_o, leader=true}
[junit4:junit4]   2> 2240608 T3602 C1292 P50337 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2240613 T3602 C1292 P50337 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3f032104 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d4e70b7),segFN=segments_1,generation=1}
[junit4:junit4]   2> 2240614 T3602 C1292 P50337 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 2240615 T3602 C1292 P50337 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3f032104 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d4e70b7),segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3f032104 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4d4e70b7),segFN=segments_2,generation=2}
[junit4:junit4]   2> 2240615 T3602 C1292 P50337 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 2240616 T3602 C1292 P50337 oass.SolrIndexSearcher.<init> Opening Searcher@4a446f77 main
[junit4:junit4]   2> 2240617 T3602 C1292 P50337 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2240618 T3611 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a446f77 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2240619 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4]   2> 2240648 T3634 C1291 P50344 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2240651 T3634 C1291 P50344 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e77aeee lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c957739),segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e77aeee lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c957739),segFN=segments_3,generation=3}
[junit4:junit4]   2> 2240652 T3634 C1291 P50344 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 2240653 T3634 C1291 P50344 oass.SolrIndexSearcher.<init> Opening Searcher@66fa2c99 main
[junit4:junit4]   2> 2240653 T3634 C1291 P50344 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2240654 T3645 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@66fa2c99 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2240655 T3634 C1291 P50344 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:50341/m_o/collection1/, StdNode: http://127.0.0.1:50347/m_o/collection1/, StdNode: http://127.0.0.1:50351/m_o/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 2240669 T3621 C1289 P50341 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2>  C1290_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50351_m_o, base_url=http://127.0.0.1:50351/m_o}
[junit4:junit4]   2> 2240679 T3669 C1290 P50351 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2240684 T3669 C1290 P50351 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@630785c8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c0e8559),segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@630785c8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c0e8559),segFN=segments_3,generation=3}
[junit4:junit4]   2> 2240684 T3669 C1290 P50351 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 2240686 T3669 C1290 P50351 oass.SolrIndexSearcher.<init> Opening Searcher@23457ba2 main
[junit4:junit4]   2> 2240686 T3669 C1290 P50351 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2240687 T3621 C1289 P50341 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@10a19980 lockFactory=org.apache.lucene.store.NativeFSLockFactory@732e2e1e),segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@10a19980 lockFactory=org.apache.lucene.store.NativeFSLockFactory@732e2e1e),segFN=segments_3,generation=3}
[junit4:junit4]   2> 2240688 T3621 C1289 P50341 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 2240688 T3621 C1289 P50341 oass.SolrIndexSearcher.<init> Opening Searcher@506b5e03 main
[junit4:junit4]   2> 2240689 T3621 C1289 P50341 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2240690 T3629 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@506b5e03 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2240691 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 22
[junit4:junit4]   2>  C1288_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50347_m_o, base_url=http://127.0.0.1:50347/m_o}
[junit4:junit4]   2> 2240693 T3653 C1288 P50347 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 2240693 T3680 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23457ba2 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2240695 T3669 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 17
[junit4:junit4]   2> 2240696 T3653 C1288 P50347 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6baff476 lockFactory=org.apache.lucene.store.NativeFSLockFactory@521dfe59),segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6baff476 lockFactory=org.apache.lucene.store.NativeFSLockFactory@521dfe59),segFN=segments_3,generation=3}
[junit4:junit4]   2> 2240697 T3653 C1288 P50347 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 2240699 T3653 C1288 P50347 oass.SolrIndexSearcher.<init> Opening Searcher@621c6b81 main
[junit4:junit4]   2> 2240699 T3653 C1288 P50347 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 2240700 T3661 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@621c6b81 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 2240701 T3653 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4]   2> 2240703 T3634 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 55
[junit4:junit4]   2> 2240706 T3585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2240711 T3622 C1289 P50341 oasc.SolrCore.execute [collection1] webapp=/m_o path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 2240717 T3654 C1288 P50347 oasc.SolrCore.execute [collection1] webapp=/m_o path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 2240721 T3636 C1291 P50344 oasc.SolrCore.execute [collection1] webapp=/m_o path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 2240728 T3670 C1290 P50351 oasc.SolrCore.execute [collection1] webapp=/m_o path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 2242746 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435703679330025472)} 0 2
[junit4:junit4]   2> 2242772 T3669 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={update.distrib=FROMLEADER&_version_=-1435703679339462656&update.from=http://127.0.0.1:50344/m_o/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435703679339462656)} 0 5
[junit4:junit4]   2> 2242775 T3654 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={update.distrib=FROMLEADER&_version_=-1435703679343656960&update.from=http://127.0.0.1:50341/m_o/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435703679343656960)} 0 5
[junit4:junit4]   2> 2242776 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1435703679343656960)} 0 18
[junit4:junit4]   2> 2242777 T3636 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435703679339462656)} 0 24
[junit4:junit4]   2> 2242783 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[0 (1435703679367774208)]} 0 2
[junit4:junit4]   2> 2242798 T3655 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1435703679379308544)]} 0 2
[junit4:junit4]   2> 2242800 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1435703679379308544)]} 0 9
[junit4:junit4]   2> 2242801 T3637 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[0]} 0 13
[junit4:junit4]   2> 2242807 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[1 (1435703679395037184)]} 0 1
[junit4:junit4]   2> 2242824 T3650 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1435703679406571520)]} 0 1
[junit4:junit4]   2> 2242825 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1435703679406571520)]} 0 8
[junit4:junit4]   2> 2242826 T3635 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[1]} 0 15
[junit4:junit4]   2> 2242830 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[2 (1435703679419154432)]} 0 0
[junit4:junit4]   2> 2242844 T3670 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1435703679427543040)]} 0 2
[junit4:junit4]   2> 2242845 T3638 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[2 (1435703679427543040)]} 0 7
[junit4:junit4]   2> 2242846 T3620 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[2]} 0 11
[junit4:junit4]   2> 2242850 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[3 (1435703679440125952)]} 0 1
[junit4:junit4]   2> 2242859 T3671 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1435703679446417408)]} 0 0
[junit4:junit4]   2> 2242860 T3639 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[3 (1435703679446417408)]} 0 4
[junit4:junit4]   2> 2242862 T3618 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[3]} 0 9
[junit4:junit4]   2> 2242865 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[4 (1435703679455854592)]} 0 0
[junit4:junit4]   2> 2242881 T3651 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1435703679466340352)]} 0 0
[junit4:junit4]   2> 2242888 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1435703679466340352)]} 0 14
[junit4:junit4]   2> 2242889 T3634 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[4]} 0 20
[junit4:junit4]   2> 2242893 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[5 (1435703679485214720)]} 0 0
[junit4:junit4]   2> 2242903 T3672 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1435703679492554752)]} 0 0
[junit4:junit4]   2> 2242905 T3636 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[5 (1435703679492554752)]} 0 5
[junit4:junit4]   2> 2242910 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[6 (1435703679501991936)]} 0 1
[junit4:junit4]   2> 2242919 T3667 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1435703679508283392)]} 0 1
[junit4:junit4]   2> 2242921 T3637 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[6 (1435703679508283392)]} 0 6
[junit4:junit4]   2> 2242926 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[7 (1435703679519817728)]} 0 0
[junit4:junit4]   2> 2242934 T3668 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1435703679524012032)]} 0 1
[junit4:junit4]   2> 2242935 T3635 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[7 (1435703679524012032)]} 0 5
[junit4:junit4]   2> 2242940 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[8 (1435703679534497792)]} 0 0
[junit4:junit4]   2> 2242952 T3652 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1435703679539740672)]} 0 2
[junit4:junit4]   2> 2242956 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[8 (1435703679539740672)]} 0 12
[junit4:junit4]   2> 2242962 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[9 (1435703679556517888)]} 0 1
[junit4:junit4]   2> 2242980 T3669 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1435703679568052224)]} 0 2
[junit4:junit4]   2> 2242982 T3638 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[9 (1435703679568052224)]} 0 10
[junit4:junit4]   2> 2242983 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[9]} 0 14
[junit4:junit4]   2> 2242987 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[10 (1435703679583780864)]} 0 1
[junit4:junit4]   2> 2242995 T3653 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1435703679587975168)]} 0 1
[junit4:junit4]   2> 2242997 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[10 (1435703679587975168)]} 0 6
[junit4:junit4]   2> 2243001 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[11 (1435703679598460928)]} 0 1
[junit4:junit4]   2> 2243008 T3654 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1435703679601606656)]} 0 1
[junit4:junit4]   2> 2243009 T3620 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[11 (1435703679601606656)]} 0 5
[junit4:junit4]   2> 2243012 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[12 (1435703679609995264)]} 0 0
[junit4:junit4]   2> 2243020 T3655 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1435703679615238144)]} 0 1
[junit4:junit4]   2> 2243022 T3618 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[12 (1435703679615238144)]} 0 7
[junit4:junit4]   2> 2243025 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[13 (1435703679623626752)]} 0 0
[junit4:junit4]   2> 2243034 T3650 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1435703679629918208)]} 0 1
[junit4:junit4]   2> 2243036 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[13 (1435703679629918208)]} 0 5
[junit4:junit4]   2> 2243037 T3639 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[13]} 0 9
[junit4:junit4]   2> 2243041 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[14 (1435703679640403968)]} 0 0
[junit4:junit4]   2> 2243053 T3651 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1435703679647744000)]} 0 1
[junit4:junit4]   2> 2243059 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[14 (1435703679647744000)]} 0 11
[junit4:junit4]   2> 2243061 T3634 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[14]} 0 16
[junit4:junit4]   2> 2243066 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[15 (1435703679665569792)]} 0 1
[junit4:junit4]   2> 2243078 T3652 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1435703679675006976)]} 0 1
[junit4:junit4]   2> 2243079 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[15 (1435703679675006976)]} 0 5
[junit4:junit4]   2> 2243081 T3636 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[15]} 0 10
[junit4:junit4]   2> 2243086 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[16 (1435703679686541312)]} 0 1
[junit4:junit4]   2> 2243099 T3653 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1435703679697027072)]} 0 1
[junit4:junit4]   2> 2243101 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[16 (1435703679697027072)]} 0 7
[junit4:junit4]   2> 2243102 T3637 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[16]} 0 11
[junit4:junit4]   2> 2243106 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[17 (1435703679708561408)]} 0 1
[junit4:junit4]   2> 2243115 T3670 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1435703679712755712)]} 0 0
[junit4:junit4]   2> 2243117 T3635 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[17 (1435703679712755712)]} 0 7
[junit4:junit4]   2> 2243120 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[18 (1435703679723241472)]} 0 1
[junit4:junit4]   2> 2243130 T3671 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1435703679730581504)]} 0 1
[junit4:junit4]   2> 2243131 T3638 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[18 (1435703679730581504)]} 0 5
[junit4:junit4]   2> 2243132 T3620 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[18]} 0 8
[junit4:junit4]   2> 2243136 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[19 (1435703679740018688)]} 0 1
[junit4:junit4]   2> 2243146 T3672 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1435703679746310144)]} 0 0
[junit4:junit4]   2> 2243148 T3634 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[19 (1435703679746310144)]} 0 6
[junit4:junit4]   2> 2243149 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[19]} 0 10
[junit4:junit4]   2> 2243159 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[20 (1435703679763087360)]} 0 2
[junit4:junit4]   2> 2243172 T3654 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1435703679773573120)]} 0 1
[junit4:junit4]   2> 2243174 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[20 (1435703679773573120)]} 0 6
[junit4:junit4]   2> 2243175 T3636 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[20]} 0 11
[junit4:junit4]   2> 2243184 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[21 (1435703679790350336)]} 0 1
[junit4:junit4]   2> 2243197 T3668 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1435703679799787520)]} 0 0
[junit4:junit4]   2> 2243198 T3637 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[21 (1435703679799787520)]} 0 5
[junit4:junit4]   2> 2243199 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[21]} 0 9
[junit4:junit4]   2> 2243204 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[22 (1435703679811321856)]} 0 1
[junit4:junit4]   2> 2243210 T3650 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1435703679814467584)]} 0 0
[junit4:junit4]   2> 2243212 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[22 (1435703679814467584)]} 0 5
[junit4:junit4]   2> 2243216 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[23 (1435703679823904768)]} 0 1
[junit4:junit4]   2> 2243223 T3669 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1435703679827050496)]} 0 1
[junit4:junit4]   2> 2243224 T3635 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[23 (1435703679827050496)]} 0 5
[junit4:junit4]   2> 2243227 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[24 (1435703679835439104)]} 0 0
[junit4:junit4]   2> 2243235 T3651 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1435703679839633408)]} 0 1
[junit4:junit4]   2> 2243237 T3618 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[24 (1435703679839633408)]} 0 6
[junit4:junit4]   2> 2243241 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[25 (1435703679850119168)]} 0 0
[junit4:junit4]   2> 2243250 T3670 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1435703679856410624)]} 0 0
[junit4:junit4]   2> 2243252 T3638 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[25 (1435703679856410624)]} 0 5
[junit4:junit4]   2> 2243253 T3620 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[25]} 0 9
[junit4:junit4]   2> 2243264 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[26 (1435703679870042112)]} 0 6
[junit4:junit4]   2> 2243271 T3671 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1435703679878430720)]} 0 0
[junit4:junit4]   2> 2243272 T3639 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[26 (1435703679878430720)]} 0 4
[junit4:junit4]   2> 2243279 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[27 (1435703679888916480)]} 0 1
[junit4:junit4]   2> 2243286 T3652 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1435703679893110784)]} 0 1
[junit4:junit4]   2> 2243287 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[27 (1435703679893110784)]} 0 5
[junit4:junit4]   2> 2243290 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[28 (1435703679901499392)]} 0 0
[junit4:junit4]   2> 2243300 T3653 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1435703679908839424)]} 0 1
[junit4:junit4]   2> 2243301 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[28 (1435703679908839424)]} 0 4
[junit4:junit4]   2> 2243303 T3634 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[28]} 0 9
[junit4:junit4]   2> 2243307 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[29 (1435703679919325184)]} 0 0
[junit4:junit4]   2> 2243314 T3667 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1435703679923519488)]} 0 1
[junit4:junit4]   2> 2243315 T3636 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[29 (1435703679923519488)]} 0 4
[junit4:junit4]   2> 2243319 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[30 (1435703679931908096)]} 0 0
[junit4:junit4]   2> 2243326 T3672 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1435703679936102400)]} 0 0
[junit4:junit4]   2> 2243334 T3637 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[30 (1435703679936102400)]} 0 11
[junit4:junit4]   2> 2243338 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[31 (1435703679951831040)]} 0 0
[junit4:junit4]   2> 2243348 T3668 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1435703679959171072)]} 0 0
[junit4:junit4]   2> 2243349 T3635 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[31 (1435703679959171072)]} 0 4
[junit4:junit4]   2> 2243356 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[32 (1435703679970705408)]} 0 1
[junit4:junit4]   2> 2243362 T3655 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1435703679973851136)]} 0 0
[junit4:junit4]   2> 2243367 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[32 (1435703679973851136)]} 0 8
[junit4:junit4]   2> 2243381 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[33 (1435703679996919808)]} 0 1
[junit4:junit4]   2> 2243397 T3669 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1435703680007405568)]} 0 1
[junit4:junit4]   2> 2243402 T3638 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[33 (1435703680007405568)]} 0 11
[junit4:junit4]   2> 2243405 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[33]} 0 17
[junit4:junit4]   2> 2243413 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[34 (1435703680029425664)]} 0 1
[junit4:junit4]   2> 2243420 T3654 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1435703680034668544)]} 0 0
[junit4:junit4]   2> 2243421 T3618 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[34 (1435703680034668544)]} 0 4
[junit4:junit4]   2> 2243426 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[35 (1435703680043057152)]} 0 1
[junit4:junit4]   2> 2243433 T3650 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1435703680048300032)]} 0 1
[junit4:junit4]   2> 2243434 T3620 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[35 (1435703680048300032)]} 0 5
[junit4:junit4]   2> 2243438 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[36 (1435703680056688640)]} 0 0
[junit4:junit4]   2> 2243445 T3651 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1435703680060882944)]} 0 0
[junit4:junit4]   2> 2243447 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[36 (1435703680060882944)]} 0 4
[junit4:junit4]   2> 2243450 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[37 (1435703680069271552)]} 0 0
[junit4:junit4]   2> 2243458 T3670 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1435703680074514432)]} 0 1
[junit4:junit4]   2> 2243459 T3639 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[37 (1435703680074514432)]} 0 5
[junit4:junit4]   2> 2243463 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[38 (1435703680082903040)]} 0 0
[junit4:junit4]   2> 2243474 T3652 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1435703680090243072)]} 0 1
[junit4:junit4]   2> 2243475 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[38 (1435703680090243072)]} 0 5
[junit4:junit4]   2> 2243476 T3634 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[38]} 0 9
[junit4:junit4]   2> 2243486 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[39 (1435703680105971712)]} 0 3
[junit4:junit4]   2> 2243493 T3671 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1435703680111214592)]} 0 0
[junit4:junit4]   2> 2243495 T3636 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[39 (1435703680111214592)]} 0 5
[junit4:junit4]   2> 2243500 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[40 (1435703680121700352)]} 0 1
[junit4:junit4]   2> 2243507 T3653 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1435703680125894656)]} 0 0
[junit4:junit4]   2> 2243509 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[40 (1435703680125894656)]} 0 5
[junit4:junit4]   2> 2243513 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[41 (1435703680134283264)]} 0 1
[junit4:junit4]   2> 2243523 T3655 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1435703680141623296)]} 0 1
[junit4:junit4]   2> 2243524 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[41 (1435703680141623296)]} 0 5
[junit4:junit4]   2> 2243526 T3637 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[41]} 0 9
[junit4:junit4]   2> 2243529 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[42 (1435703680152109056)]} 0 0
[junit4:junit4]   2> 2243539 T3654 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1435703680159449088)]} 0 0
[junit4:junit4]   2> 2243541 T3620 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[42 (1435703680159449088)]} 0 5
[junit4:junit4]   2> 2243542 T3635 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[42]} 0 9
[junit4:junit4]   2> 2243546 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[43 (1435703680168886272)]} 0 1
[junit4:junit4]   2> 2243553 T3650 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1435703680174129152)]} 0 0
[junit4:junit4]   2> 2243555 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[43 (1435703680174129152)]} 0 5
[junit4:junit4]   2> 2243565 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[44 (1435703680189857792)]} 0 1
[junit4:junit4]   2> 2243576 T3651 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1435703680197197824)]} 0 1
[junit4:junit4]   2> 2243577 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[44 (1435703680197197824)]} 0 5
[junit4:junit4]   2> 2243578 T3639 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[44]} 0 9
[junit4:junit4]   2> 2243582 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[45 (1435703680207683584)]} 0 1
[junit4:junit4]   2> 2243589 T3667 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1435703680211877888)]} 0 0
[junit4:junit4]   2> 2243591 T3634 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[45 (1435703680211877888)]} 0 6
[junit4:junit4]   2> 2243595 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[46 (1435703680220266496)]} 0 1
[junit4:junit4]   2> 2243604 T3653 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1435703680226557952)]} 0 1
[junit4:junit4]   2> 2243606 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[46 (1435703680226557952)]} 0 6
[junit4:junit4]   2> 2243610 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[47 (1435703680235995136)]} 0 1
[junit4:junit4]   2> 2243619 T3655 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1435703680243335168)]} 0 0
[junit4:junit4]   2> 2243621 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[47 (1435703680243335168)]} 0 5
[junit4:junit4]   2> 2243622 T3636 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[47]} 0 9
[junit4:junit4]   2> 2243627 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[48 (1435703680253820928)]} 0 2
[junit4:junit4]   2> 2243633 T3654 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1435703680258015232)]} 0 0
[junit4:junit4]   2> 2243634 T3618 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[48 (1435703680258015232)]} 0 4
[junit4:junit4]   2> 2243638 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[49 (1435703680266403840)]} 0 1
[junit4:junit4]   2> 2243647 T3650 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1435703680272695296)]} 0 1
[junit4:junit4]   2> 2243648 T3620 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[49 (1435703680272695296)]} 0 4
[junit4:junit4]   2> 2243649 T3637 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[49]} 0 8
[junit4:junit4]   2> 2243652 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[50 (1435703680281083904)]} 0 0
[junit4:junit4]   2> 2243661 T3652 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1435703680287375360)]} 0 0
[junit4:junit4]   2> 2243663 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[50 (1435703680287375360)]} 0 5
[junit4:junit4]   2> 2243667 T3638 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[50]} 0 12
[junit4:junit4]   2> 2243679 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[51 (1435703680309395456)]} 0 1
[junit4:junit4]   2> 2243693 T3651 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1435703680318832640)]} 0 1
[junit4:junit4]   2> 2243695 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[51 (1435703680318832640)]} 0 7
[junit4:junit4]   2> 2243696 T3635 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[51]} 0 12
[junit4:junit4]   2> 2243701 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[52 (1435703680331415552)]} 0 1
[junit4:junit4]   2> 2243712 T3653 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1435703680338755584)]} 0 1
[junit4:junit4]   2> 2243714 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[52 (1435703680338755584)]} 0 7
[junit4:junit4]   2> 2243715 T3639 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[52]} 0 11
[junit4:junit4]   2> 2243718 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[53 (1435703680350289920)]} 0 0
[junit4:junit4]   2> 2243726 T3668 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1435703680355532800)]} 0 0
[junit4:junit4]   2> 2243727 T3634 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[53 (1435703680355532800)]} 0 4
[junit4:junit4]   2> 2243728 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[53]} 0 7
[junit4:junit4]   2> 2243732 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[54 (1435703680364969984)]} 0 1
[junit4:junit4]   2> 2243742 T3655 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1435703680371261440)]} 0 0
[junit4:junit4]   2> 2243744 T3618 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[54 (1435703680371261440)]} 0 6
[junit4:junit4]   2> 2243745 T3636 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[54]} 0 10
[junit4:junit4]   2> 2243748 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[55 (1435703680381747200)]} 0 0
[junit4:junit4]   2> 2243763 T3654 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1435703680394330112)]} 0 1
[junit4:junit4]   2> 2243764 T3620 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[55 (1435703680394330112)]} 0 5
[junit4:junit4]   2> 2243765 T3637 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[55]} 0 8
[junit4:junit4]   2> 2243769 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[56 (1435703680403767296)]} 0 0
[junit4:junit4]   2> 2243786 T3669 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1435703680417398784)]} 0 0
[junit4:junit4]   2> 2243788 T3638 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[56 (1435703680417398784)]} 0 7
[junit4:junit4]   2> 2243791 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[56]} 0 15
[junit4:junit4]   2> 2243797 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[57 (1435703680433127424)]} 0 1
[junit4:junit4]   2> 2243811 T3670 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1435703680443613184)]} 0 0
[junit4:junit4]   2> 2243813 T3635 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[57 (1435703680443613184)]} 0 6
[junit4:junit4]   2> 2243814 T3622 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[57]} 0 11
[junit4:junit4]   2> 2243819 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[58 (1435703680455147520)]} 0 1
[junit4:junit4]   2> 2243828 T3671 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1435703680461438976)]} 0 1
[junit4:junit4]   2> 2243831 T3639 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[58 (1435703680461438976)]} 0 8
[junit4:junit4]   2> 2243836 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[59 (1435703680472973312)]} 0 0
[junit4:junit4]   2> 2243846 T3650 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1435703680480313344)]} 0 1
[junit4:junit4]   2> 2243848 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[59 (1435703680480313344)]} 0 6
[junit4:junit4]   2> 2243849 T3634 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[59]} 0 10
[junit4:junit4]   2> 2243862 T3599 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[60 (1435703680500236288)]} 0 1
[junit4:junit4]   2> 2243879 T3672 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1435703680511770624)]} 0 0
[junit4:junit4]   2> 2243880 T3636 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[60 (1435703680511770624)]} 0 9
[junit4:junit4]   2> 2243881 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[60]} 0 15
[junit4:junit4]   2> 2243885 T3601 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[61 (1435703680525402112)]} 0 1
[junit4:junit4]   2> 2243894 T3652 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[61 (1435703680530644992)]} 0 1
[junit4:junit4]   2> 2243896 T3618 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[61 (1435703680530644992)]} 0 6
[junit4:junit4]   2> 2243901 T3602 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[62 (1435703680541130752)]} 0 1
[junit4:junit4]   2> 2243911 T3667 C1290 P50351 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[62 (1435703680548470784)]} 0 1
[junit4:junit4]   2> 2243912 T3637 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[62 (1435703680548470784)]} 0 5
[junit4:junit4]   2> 2243914 T3620 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[62]} 0 9
[junit4:junit4]   2> 2243918 T3597 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[63 (1435703680558956544)]} 0 1
[junit4:junit4]   2> 2243925 T3651 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[63 (1435703680563150848)]} 0 1
[junit4:junit4]   2> 2243927 T3621 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[63 (1435703680563150848)]} 0 6
[junit4:junit4]   2> 2243932 T3598 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[64 (1435703680574685184)]} 0 1
[junit4:junit4]   2> 2243943 T3653 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[64 (1435703680583073792)]} 0 0
[junit4:junit4]   2> 2243945 T3623 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[64 (1435703680583073792)]} 0 6
[junit4:junit4]   2> 2243946 T3638 C1291 P50344 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[64]} 0 10
[junit4:junit4]   2> 2243950 T3600 C1292 P50337 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={wt=javabin&version=2} {add=[65 (1435703680593559552)]} 0 1
[junit4:junit4]   2> 2243966 T3655 C1288 P50347 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50341/m_o/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[65 (1435703680601948160)]} 0 1
[junit4:junit4]   2> 2243968 T3619 C1289 P50341 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_o path=/update params={distrib.from=http://127.0.0.1:50344/m_o/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[65 (1435703680601948160)]} 0 11
[junit4:j

[...truncated too long message...]

ode_name":"127.0.0.1:50351_m_o",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50351/m_o"}
[junit4:junit4]   2> 2280569 T3676 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2281548 T3585 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2281549 T3585 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2281555 T3585 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@42c25f65
[junit4:junit4]   2> 2281556 T3585 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=248,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 2281557 T3585 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 2281558 T3585 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 2281558 T3585 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 2281562 T3585 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 2281563 T3585 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4]   2> 2281563 T3585 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4;done=false>>]
[junit4:junit4]   2> 2281564 T3585 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4
[junit4:junit4]   2> 2281577 T3585 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index;done=false>>]
[junit4:junit4]   2> 2281578 T3585 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260/jetty4/index
[junit4:junit4]   2> 2281579 T3737 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89731477686779916-127.0.0.1:50351_m_o-n_0000000004) am no longer a leader.
[junit4:junit4]   2> 2281608 T3585 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/m_o,null}
[junit4:junit4]   2> 2281699 T3585 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 2281704 T3585 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50334 50334
[junit4:junit4]   2> 2281809 T3585 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50337
[junit4:junit4]   2> !!!! WARNING: best effort to remove /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369193690260 FAILED !!!!!
[junit4:junit4]   2> 2281811 T3585 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 2281811 T3585 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50334 50334
[junit4:junit4]   1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=A7CD213BFA1FDE57 -Dtests.slow=true -Dtests.locale=hr_HR -Dtests.timezone=CST6CDT -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR   73.8s | ChaosMonkeyShardSplitTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:50341/m_o returned non ok status:500, message:Server Error
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([A7CD213BFA1FDE57:262BAF238D40BE6B]:0)
[junit4:junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4]    > 	at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
[junit4:junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 2281858 T3585 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 73817 T3584 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4]   2> 2282894 T3676 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 2282895 T3676 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2282895 T3676 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=hr_HR, timezone=CST6CDT
[junit4:junit4]   2> NOTE: Mac OS X 10.8.3 x86_64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=2,threads=1,free=100542248,total=378994688
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestPartialUpdateDeduplication, TestPluginEnable, TestDFRSimilarityFactory, TestPostingsSolrHighlighter, UpdateParamsTest, TestStressRecovery, ShardRoutingTest, TestSolrXml, AnalysisAfterCoreReloadTest, ZkSolrClientTest, MultiTermTest, PrimUtilsTest, DateFieldTest, TestCloudManagedSchemaAddField, TestLFUCache, TestFieldTypeCollectionResource, TestQuerySenderNoQuery, TestSolrJ, TestSolrCoreProperties, DirectUpdateHandlerOptimizeTest, TestTrie, ZkCLITest, TestWriterPerf, RequestHandlersTest, FileUtilsTest, ShardSplitTest, IndexSchemaRuntimeFieldTest, FieldAnalysisRequestHandlerTest, TestFuzzyAnalyzedSuggestions, TestJmxIntegration, IndexBasedSpellCheckerTest, DOMUtilTest, TestDistributedGrouping, ReturnFieldsTest, PolyFieldTest, DocumentBuilderTest, LegacyHTMLStripCharFilterTest, ShowFileRequestHandlerTest, CircularListTest, TestSurroundQueryParser, FastVectorHighlighterTest, SolrCoreTest, DistributedSpellCheckComponentTest, SpellCheckCollatorTest, QueryEqualityTest, FieldMutatingUpdateProcessorTest, CurrencyFieldXmlFileTest, TestRecovery, TestAddFieldRealTimeGet, TestClassNameShortening, TestBM25SimilarityFactory, TestSolrQueryParserDefaultOperatorResource, PluginInfoTest, TestQuerySenderListener, SolrIndexConfigTest, HighlighterConfigTest, JsonLoaderTest, TestJmxMonitoredMap, TestReversedWildcardFilterFactory, ZkNodePropsTest, TestRealTimeGet, ClusterStateUpdateTest, ShardRoutingCustomTest, TestQueryTypes, QueryParsingTest, MinimalSchemaTest, TestRandomDVFaceting, PreAnalyzedUpdateProcessorTest, EchoParamsTest, SortByFunctionTest, TestUniqueKeyFieldResource, OpenCloseCoreStressTest, IndexSchemaTest, HardAutoCommitTest, TestSystemIdResolver, TestAnalyzedSuggestions, FileBasedSpellCheckerTest, ResourceLoaderTest, NoCacheHeaderTest, TestSuggestSpellingConverter, DirectSolrConnectionTest, TestMaxScoreQueryParser, SpellCheckComponentTest, PreAnalyzedFieldTest, TestElisionMultitermQuery, TestJoin, TestCSVResponseWriter, BasicFunctionalityTest, TestWordDelimiterFilterFactory, LeaderElectionIntegrationTest, ChaosMonkeySafeLeaderTest, DirectUpdateHandlerTest, CollectionsAPIDistributedZkTest, TestStressVersions, TestSchemaVersionResource, OpenExchangeRatesOrgProviderTest, SolrInfoMBeanTest, TermVectorComponentTest, TermsComponentTest, RegexBoostProcessorTest, SynonymTokenizerTest, SyncSliceTest, SolrIndexSplitterTest, TestDynamicFieldResource, SolrCmdDistributorTest, CacheHeaderTest, FullSolrCloudDistribCmdsTest, ExternalFileFieldSortTest, DirectSolrSpellCheckerTest, TestSchemaSimilarityResource, SignatureUpdateProcessorFactoryTest, TestFoldingMultitermQuery, LukeRequestHandlerTest, StatelessScriptUpdateProcessorFactoryTest, PingRequestHandlerTest, CSVRequestHandlerTest, SolrRequestParserTest, TestFieldCollectionResource, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed in 75.14s, 1 test, 1 error <<< FAILURES!

[...truncated 531 lines...]
BUILD FAILED
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:380: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:360: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:181: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:437: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1240: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:884: There were test failures: 295 suites, 1231 tests, 1 error, 18 ignored (6 assumptions)

Total time: 109 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure