You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/24 18:38:26 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-fcs-b129) - Build # 9584 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9584/
Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Could not find collection:addReplicaColl

Stack Trace:
java.lang.AssertionError: Could not find collection:addReplicaColl
	at __randomizedtesting.SeedInfo.seed([D2142FDC4C381343:53F2A1C43B67737F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:147)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:724)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1193)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10415 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 570747 T1579 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 570747 T1579 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /a/rn
   [junit4]   2> 570751 T1579 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1393262608233
   [junit4]   2> 570752 T1579 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 570752 T1580 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 570852 T1579 oasc.ZkTestServer.run start zk server on port:50322
   [junit4]   2> 570853 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 570877 T1586 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13ed7b name:ZooKeeperConnection Watcher:127.0.0.1:50322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 570878 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 570878 T1579 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 570882 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 570883 T1588 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13c2441 name:ZooKeeperConnection Watcher:127.0.0.1:50322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 570883 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 570883 T1579 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 570885 T1579 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 570887 T1579 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 570888 T1579 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 570890 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 570890 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 570893 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 570893 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 570895 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 570895 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 570897 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 570897 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 570899 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 570899 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 570901 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 570901 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 570903 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 570903 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 570908 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 570908 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 570910 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 570911 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 570912 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 570913 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 570914 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 570914 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 570917 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 570917 T1590 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@113628 name:ZooKeeperConnection Watcher:127.0.0.1:50322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 570918 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 570919 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 570920 T1592 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8bded0 name:ZooKeeperConnection Watcher:127.0.0.1:50322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 570920 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 570920 T1579 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 570922 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 570923 T1594 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@104868f name:ZooKeeperConnection Watcher:127.0.0.1:50322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 570923 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 570924 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 570924 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 570927 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 570927 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 570929 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 570929 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 570931 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 570931 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 570933 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 570933 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 570934 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 570935 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 570937 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 570937 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 570939 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 570939 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 570941 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 570941 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 570943 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 570943 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 570944 T1579 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 570944 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 571166 T1579 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 571169 T1579 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 571174 T1579 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53093
   [junit4]   2> 571175 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 571176 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 571176 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428
   [junit4]   2> 571176 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/'
   [junit4]   2> 571186 T1579 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/solr.xml
   [junit4]   2> 571192 T1579 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428
   [junit4]   2> 571192 T1579 oasc.CoreContainer.<init> New CoreContainer 22888754
   [junit4]   2> 571193 T1579 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/]
   [junit4]   2> 571193 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 571193 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 571194 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 571194 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 571194 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 571194 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 571195 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 571195 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 571195 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 571199 T1579 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 571199 T1579 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 571199 T1579 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 571200 T1579 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50322/solr
   [junit4]   2> 571200 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 571201 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 571202 T1605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f7033c name:ZooKeeperConnection Watcher:127.0.0.1:50322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 571203 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 571204 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 571205 T1607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b6c7d6 name:ZooKeeperConnection Watcher:127.0.0.1:50322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 571205 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 571207 T1579 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 571209 T1579 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 571211 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 571212 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53093_a%2Frn
   [junit4]   2> 571213 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53093_a%2Frn
   [junit4]   2> 571215 T1579 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 571217 T1579 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 571219 T1579 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53093_a%2Frn
   [junit4]   2> 571219 T1579 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 571220 T1579 oasc.Overseer.start Overseer (id=91308858293354502-127.0.0.1:53093_a%2Frn-n_0000000000) starting
   [junit4]   2> 571222 T1579 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 571225 T1609 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 571225 T1579 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 571226 T1609 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 571227 T1579 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 571228 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 571229 T1608 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 571229 T1579 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428
   [junit4]   2> 571237 T1579 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/conf/
   [junit4]   2> 571357 T1579 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 571358 T1579 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 571358 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 571361 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 571362 T1612 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f88a7c name:ZooKeeperConnection Watcher:127.0.0.1:50322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 571362 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 571363 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 571364 T1579 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 571571 T1579 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 571572 T1579 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 571574 T1579 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 571575 T1579 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60799
   [junit4]   2> 571577 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 571577 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 571577 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393262608846
   [junit4]   2> 571577 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393262608846/'
   [junit4]   2> 571586 T1579 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393262608846/solr.xml
   [junit4]   2> 571592 T1579 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393262608846
   [junit4]   2> 571593 T1579 oasc.CoreContainer.<init> New CoreContainer 1500955
   [junit4]   2> 571593 T1579 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393262608846/]
   [junit4]   2> 571593 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 571593 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 571594 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 571594 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 571594 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 571595 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 571595 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 571595 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 571595 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 571598 T1579 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 571598 T1579 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 571599 T1579 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 571599 T1579 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50322/solr
   [junit4]   2> 571599 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 571600 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 571601 T1623 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44dd42 name:ZooKeeperConnection Watcher:127.0.0.1:50322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 571602 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 571604 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 571605 T1625 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@165ad45 name:ZooKeeperConnection Watcher:127.0.0.1:50322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 571605 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 571607 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 572609 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60799_a%2Frn
   [junit4]   2> 572610 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60799_a%2Frn
   [junit4]   2> 572612 T1612 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 572612 T1607 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 572612 T1625 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 572615 T1579 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393262608846
   [junit4]   2> 572622 T1579 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393262608846/conf/
   [junit4]   2> 572707 T1579 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 572707 T1579 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 572708 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 572931 T1579 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 572932 T1579 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 572935 T1579 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 572935 T1579 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:40540
   [junit4]   2> 572938 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 572939 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 572939 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191
   [junit4]   2> 572940 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/'
   [junit4]   2> 572955 T1579 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/solr.xml
   [junit4]   2> 572960 T1579 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191
   [junit4]   2> 572961 T1579 oasc.CoreContainer.<init> New CoreContainer 20606655
   [junit4]   2> 572961 T1579 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/]
   [junit4]   2> 572962 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 572962 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 572962 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 572962 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 572963 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 572963 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 572963 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 572963 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 572964 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 572966 T1579 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 572967 T1579 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 572967 T1579 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 572967 T1579 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50322/solr
   [junit4]   2> 572967 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 572968 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 572970 T1637 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@426c98 name:ZooKeeperConnection Watcher:127.0.0.1:50322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 572971 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 572975 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 572976 T1639 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18d8384 name:ZooKeeperConnection Watcher:127.0.0.1:50322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 572976 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 572982 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 573987 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40540_a%2Frn
   [junit4]   2> 573988 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40540_a%2Frn
   [junit4]   2> 573990 T1607 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 573990 T1612 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 573990 T1625 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 573990 T1639 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 573993 T1579 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191
   [junit4]   2> 574001 T1579 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/conf/
   [junit4]   2> 574098 T1579 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 574099 T1579 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 574099 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 574303 T1579 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 574304 T1579 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 574307 T1579 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 574308 T1579 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:38517
   [junit4]   2> 574310 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 574310 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 574311 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582
   [junit4]   2> 574311 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/'
   [junit4]   2> 574321 T1579 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/solr.xml
   [junit4]   2> 574327 T1579 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582
   [junit4]   2> 574327 T1579 oasc.CoreContainer.<init> New CoreContainer 24248062
   [junit4]   2> 574328 T1579 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/]
   [junit4]   2> 574328 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 574328 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 574329 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 574329 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 574329 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 574329 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 574330 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 574330 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 574330 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 574333 T1579 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 574333 T1579 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 574334 T1579 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 574334 T1579 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50322/solr
   [junit4]   2> 574334 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 574335 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 574336 T1651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1511ea6 name:ZooKeeperConnection Watcher:127.0.0.1:50322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 574337 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 574340 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 574341 T1653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11c1350 name:ZooKeeperConnection Watcher:127.0.0.1:50322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 574341 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 574345 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 575347 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38517_a%2Frn
   [junit4]   2> 575349 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38517_a%2Frn
   [junit4]   2> 575353 T1653 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 575354 T1639 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 575354 T1607 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 575353 T1612 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 575353 T1625 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 575359 T1579 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582
   [junit4]   2> 575371 T1579 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/conf/
   [junit4]   2> 575470 T1579 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 575471 T1579 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 575471 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 575679 T1579 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 575679 T1579 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 575683 T1579 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 575685 T1579 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:46887
   [junit4]   2> 575689 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 575690 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 575690 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954
   [junit4]   2> 575691 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/'
   [junit4]   2> 575711 T1579 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/solr.xml
   [junit4]   2> 575717 T1579 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954
   [junit4]   2> 575718 T1579 oasc.CoreContainer.<init> New CoreContainer 21643781
   [junit4]   2> 575718 T1579 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/]
   [junit4]   2> 575718 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 575718 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 575719 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 575719 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 575719 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 575720 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 575720 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 575720 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 575720 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 575723 T1579 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 575723 T1579 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 575724 T1579 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 575724 T1579 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50322/solr
   [junit4]   2> 575724 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 575725 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 575727 T1665 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@112ba68 name:ZooKeeperConnection Watcher:127.0.0.1:50322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 575728 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 575731 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 575732 T1667 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35d321 name:ZooKeeperConnection Watcher:127.0.0.1:50322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 575733 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 575740 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 576743 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46887_a%2Frn
   [junit4]   2> 576745 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46887_a%2Frn
   [junit4]   2> 576749 T1612 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 576750 T1653 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 576750 T1639 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 576750 T1667 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 576750 T1625 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 576750 T1607 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 576755 T1579 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954
   [junit4]   2> 576765 T1579 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/conf/
   [junit4]   2> 576852 T1579 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 576853 T1579 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 576853 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 576948 T1620 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&router.name=compositeId&action=CREATE&numShards=2&wt=javabin&version=2
   [junit4]   2> 576949 T1607 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 576951 T1609 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 576951 T1609 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 576954 T1609 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 576954 T1609 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 576958 T1607 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 576960 T1608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 576961 T1608 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 576961 T1608 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 576963 T1607 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 576965 T1607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 576965 T1639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 576965 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 576965 T1612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 576965 T1625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 576965 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 577058 T1609 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 577059 T1609 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:38517_a%2Frn
   [junit4]   2> 577061 T1609 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:53093_a%2Frn
   [junit4]   2> 577069 T1609 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:40540_a%2Frn
   [junit4]   2> 577075 T1609 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:46887_a%2Frn
   [junit4]   2> 577127 T1600 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 577127 T1647 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 577128 T1607 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 577128 T1600 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 577128 T1647 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 577128 T1608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 577129 T1608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53093/a/rn",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53093_a%2Frn",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 577131 T1607 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 577131 T1634 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 577131 T1608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38517/a/rn",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38517_a%2Frn",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 577136 T1607 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 577136 T1634 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 577137 T1608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:40540/a/rn",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40540_a%2Frn",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 577139 T1607 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 577146 T1663 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 577147 T1607 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 577147 T1663 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 577148 T1608 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46887/a/rn",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46887_a%2Frn",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 577149 T1607 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 577252 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 577252 T1607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 577252 T1612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 577252 T1639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 577252 T1625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 577252 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 578128 T1600 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 578128 T1647 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 578130 T1600 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/solrj_collection_shard1_replica2
   [junit4]   2> 578130 T1647 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/solrj_collection_shard1_replica1
   [junit4]   2> 578130 T1647 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 578130 T1600 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 578131 T1647 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 578132 T1647 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 578132 T1600 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 578133 T1600 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 578134 T1647 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 578134 T1647 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/solrj_collection_shard1_replica1/'
   [junit4]   2> 578135 T1600 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 578135 T1600 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/solrj_collection_shard1_replica2/'
   [junit4]   2> 578136 T1634 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 578138 T1634 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/solrj_collection_shard2_replica1
   [junit4]   2> 578139 T1634 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 578140 T1634 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 578141 T1634 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 578144 T1634 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 578144 T1634 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/solrj_collection_shard2_replica1/'
   [junit4]   2> 578148 T1663 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 578151 T1663 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/solrj_collection_shard2_replica2
   [junit4]   2> 578152 T1663 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 578153 T1663 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 578153 T1663 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 578155 T1663 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 578155 T1663 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/solrj_collection_shard2_replica2/'
   [junit4]   2> 578199 T1634 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 578201 T1647 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 578205 T1600 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 578209 T1634 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 578211 T1634 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 578212 T1647 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 578212 T1663 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 578213 T1647 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 578215 T1634 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 578218 T1647 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 578218 T1600 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 578219 T1600 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 578225 T1663 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 578232 T1663 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 578233 T1600 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 578241 T1663 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 578347 T1647 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 578349 T1634 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 578352 T1647 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 578354 T1634 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 578354 T1647 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 578356 T1634 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 578367 T1647 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 578368 T1634 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 578371 T1634 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 578371 T1647 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 578373 T1600 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 578374 T1647 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 578373 T1634 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 578375 T1647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 578375 T1634 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 578378 T1634 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 578377 T1647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 578378 T1634 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 578379 T1647 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 578380 T1600 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 578380 T1634 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 578380 T1634 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 578380 T1663 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 578381 T1647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 578381 T1647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 578382 T1600 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 578381 T1634 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 578382 T1647 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 578382 T1634 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/solrj_collection_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/solrj_collection_shard2_replica1/data/
   [junit4]   2> 578383 T1647 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/solrj_collection_shard1_replica1/data/
   [junit4]   2> 578383 T1634 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1032d05
   [junit4]   2> 578383 T1647 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1032d05
   [junit4]   2> 578384 T1634 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/solrj_collection_shard2_replica1/data
   [junit4]   2> 578384 T1663 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 578385 T1634 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 578385 T1647 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/solrj_collection_shard1_replica1/data
   [junit4]   2> 578386 T1647 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 578385 T1634 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 578386 T1663 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 578386 T1647 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 578387 T1634 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393262610191/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 578387 T1647 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393262611582/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 578388 T1634 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.437130031507979]
   [junit4]   2> 578388 T1647 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.437130031507979]
   [junit4]   2> 578389 T1634 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@feb11e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@102705e),segFN=segments_1,generation=1}
   [junit4]   2> 578390 T1634 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 578390 T1647 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a3459a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1eb09de),segFN=segments_1,generation=1}
   [junit4]   2> 578391 T1647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 578392 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 578393 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 578393 T1634 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 578393 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 578393 T1647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 578393 T1634 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 578394 T1647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 578394 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 578394 T1647 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 578395 T1634 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 578395 T1647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 578395 T1634 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 578396 T1634 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 578395 T1647 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 578396 T1634 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 578396 T1647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 578397 T1634 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 578398 T1634 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 578398 T1600 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 578398 T1647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 578398 T1634 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 578399 T1647 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 578400 T1647 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 578400 T1634 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 578400 T1647 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 578401 T1600 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 578401 T1647 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 578401 T1647 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 578401 T1634 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 578402 T1647 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 578402 T1634 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 578402 T1647 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 578402 T1634 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 578403 T1647 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 578403 T1647 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 578404 T1647 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 578405 T1634 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 578405 T1647 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 578406 T1663 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 578405 T1600 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 578406 T1634 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 578407 T1600 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 578407 T1634 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 578408 T1647 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 578408 T1634 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=37, maxMergeAtOnceExplicit=35, maxMergedSegmentMB=75.42578125, floorSegmentMB=0.75390625, forceMergeDeletesPctAllowed=11.312250399806816, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 578407 T1600 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 578408 T1634 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@feb11e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@102705e),segFN=segments_1,generation=1}
   [junit4]   2> 578409 T1634 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 578408 T1663 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 578408 T1647 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 578409 T1634 oass.SolrIndexSearcher.<init> Opening Searcher@9c9c4b[solrj_collection_shard2_replica1] main
   [junit4]   2> 578409 T1600 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 578410 T1647 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=37, maxMergeAtOnceExplicit=35, maxMergedSegmentMB=75.42578125, floorSegmentMB=0.75390625, forceMergeDeletesPctAllowed=11.312250399806816, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 578411 T1647 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a3459a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1eb09de),segFN=segments_1,generation=1}
   [junit4]   2> 578411 T1647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 578411 T1600 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 578412 T1673 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica1] Registered new searcher Searcher@9c9c4b[solrj_collection_shard2_replica1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 578412 T1600 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 578411 T1647 oass.SolrIndexSearcher.<init> Opening Searcher@14f4c50[solrj_collection_shard1_replica1] main
   [junit4]   2> 578411 T1663 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 578412 T1600 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 578413 T1634 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica1
   [junit4]   2> 578413 T1600 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/solrj_collection_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/solrj_collection_shard1_replica2/data/
   [junit4]   2> 578414 T1674 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@14f4c50[solrj_collection_shard1_replica1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 578414 T1600 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1032d05
   [junit4]   2> 578413 T1634 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica1 address:https://127.0.0.1:40540/a/rn collection:solrj_collection shard:shard2
   [junit4]   2> 578414 T1663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 578415 T1663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 578415 T1647 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
   [junit4]   2> 578415 T1600 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/solrj_collection_shard1_replica2/data
   [junit4]   2> 578415 T1647 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:https://127.0.0.1:38517/a/rn collection:solrj_collection shard:shard1
   [junit4]   2> 578415 T1663 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 578416 T1634 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 578416 T1600 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 578417 T1663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 578417 T1663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 578417 T1600 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 578416 T1647 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 578418 T1600 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393262608428/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 578418 T1663 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 578419 T1600 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.437130031507979]
   [junit4]   2> 578419 T1663 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/solrj_collection_shard2_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/solrj_collection_shard2_replica2/data/
   [junit4]   2> 578420 T1600 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@29e474 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1334aee),segFN=segments_1,generation=1}
   [junit4]   2> 578420 T1600 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 578420 T1663 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1032d05
   [junit4]   2> 578421 T1634 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 578421 T1663 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/solrj_collection_shard2_replica2/data
   [junit4]   2> 578422 T1663 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 578422 T1663 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 578423 T1600 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 578423 T1663 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393262612954/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 578423 T1647 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader pro

[...truncated too long message...]

va:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 661838 T1667 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_1/leader_elect/shard3/election/91308858293354511-core_node12-n_0000000003
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 661839 T1667 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 661839 T1667 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 661840 T1667 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 661840 T1667 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene46: {other_tl1=PostingsFormat(name=Lucene41WithOrds), _version_=PostingsFormat(name=Lucene41WithOrds), multiDefault=PostingsFormat(name=Lucene41WithOrds), a_t=PostingsFormat(name=SimpleText), a_si=PostingsFormat(name=Lucene41WithOrds), intDefault=PostingsFormat(name=Lucene41WithOrds), range_facet_sl=PostingsFormat(name=MockSep), range_facet_si=PostingsFormat(name=SimpleText), id=PostingsFormat(name=MockSep), text=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), range_facet_l=PostingsFormat(name=Lucene41WithOrds), timestamp=PostingsFormat(name=Lucene41WithOrds)}, docValues:{timestamp=DocValuesFormat(name=Lucene45)}, sim=DefaultSimilarity, locale=ca_ES, timezone=America/Buenos_Aires
   [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=1,free=314916264,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestMultiCoreConfBootstrap, TestBlendedInfixSuggestions, CachingDirectoryFactoryTest, TestBinaryResponseWriter, SuggesterTSTTest, TestBinaryField, TestJmxIntegration, TestCursorMarkWithoutUniqueKey, QueryResultKeyTest, TestHashPartitioner, TestSolrXmlPersistence, TestStressReorder, XsltUpdateRequestHandlerTest, RequestHandlersTest, InfoHandlerTest, PathHierarchyTokenizerFactoryTest, TermVectorComponentDistributedTest, JSONWriterTest, StandardRequestHandlerTest, TestRangeQuery, CoreContainerCoreInitFailuresTest, MigrateRouteKeyTest, NoCacheHeaderTest, TestDefaultSimilarityFactory, TestOverriddenPrefixQueryForCustomFieldType, ShardRoutingCustomTest, SolrRequestParserTest, QueryFacetTest, TestIndexingPerformance, TestFileDictionaryLookup, DefaultValueUpdateProcessorTest, DisMaxRequestHandlerTest, TestPartialUpdateDeduplication, AbstractAnalyticsFacetTest, DebugComponentTest, RAMDirectoryFactoryTest, PreAnalyzedFieldTest, TestSolrCoreProperties, TestCoreDiscovery, DeleteReplicaTest, TestReload, TestDistribDocBasedVersion, TimeZoneUtilsTest, AnalysisErrorHandlingTest, ZkNodePropsTest, TestIBSimilarityFactory, TestManagedSchemaFieldResource, TestSolr4Spatial, TestDistributedMissingSort, BadCopyFieldTest, DistribCursorPagingTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestRecovery, TestNonNRTOpen, TestZkChroot, HdfsUnloadDistributedZkTest, TestRTGBase, SimplePostToolTest, TestSolrXml, TestUpdate, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 92.07s, 1 test, 1 failure <<< FAILURES!

[...truncated 673 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1606 tests, 1 failure, 35 ignored (7 assumptions)

Total time: 46 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9591 - Still Failing!

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
I'll fix.

On Tue, Feb 25, 2014 at 6:02 PM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9591/
> Java: 32bit/jdk1.7.0_51 -client -XX:+UseSerialGC
>
> 2 tests failed.
> REGRESSION:  org.apache.solr.core.TestNonNRTOpen.testReaderIsNotNRT
>
> Error Message:
> expected:<3> but was:<2>
>
> Stack Trace:
> java.lang.AssertionError: expected:<3> but was:<2>
>         at __randomizedtesting.SeedInfo.seed([F42EFCF46D93B00A:41A89D73D25202FE]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at org.junit.Assert.failNotEquals(Assert.java:647)
>         at org.junit.Assert.assertEquals(Assert.java:128)
>         at org.junit.Assert.assertEquals(Assert.java:472)
>         at org.junit.Assert.assertEquals(Assert.java:456)
>         at org.apache.solr.core.TestNonNRTOpen.assertNotNRT(TestNonNRTOpen.java:133)
>         at org.apache.solr.core.TestNonNRTOpen.testReaderIsNotNRT(TestNonNRTOpen.java:94)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at java.lang.Thread.run(Thread.java:744)
>
>
> FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
>
> Error Message:
> Could not find collection:addReplicaColl
>
> Stack Trace:
> java.lang.AssertionError: Could not find collection:addReplicaColl
>         at __randomizedtesting.SeedInfo.seed([F42EFCF46D93B00A:75C872EC1ACCD036]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at org.junit.Assert.assertTrue(Assert.java:43)
>         at org.junit.Assert.assertNotNull(Assert.java:526)
>         at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:147)
>         at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
>         at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
>         at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:724)
>         at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1193)
>         at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:213)
>         at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at java.lang.Thread.run(Thread.java:744)
>
>
>
>
> Build Log:
> [...truncated 10032 lines...]
>    [junit4] Suite: org.apache.solr.core.TestNonNRTOpen
>    [junit4]   2> 211034 T669 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
>    [junit4]   2> 211035 T669 oas.SolrTestCaseJ4.initCore ####initCore
>    [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238
>    [junit4]   2> 211035 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
>    [junit4]   2> 211036 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
>    [junit4]   2> 211036 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
>    [junit4]   2> 211036 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
>    [junit4]   2> 211074 T669 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>    [junit4]   2> 211111 T669 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
>    [junit4]   2> 211111 T669 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
>    [junit4]   2> 211115 T669 oass.IndexSchema.readSchema [null] Schema name=minimal
>    [junit4]   2> 211128 T669 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
>    [junit4]   2> 211130 T669 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>    [junit4]   2> 211130 T669 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
>    [junit4]   2> 211130 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
>    [junit4]   2> 211177 T669 oasc.CoreContainer.<init> New CoreContainer 19970511
>    [junit4]   2> 211178 T669 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
>    [junit4]   2> 211179 T669 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
>    [junit4]   2> 211179 T669 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
>    [junit4]   2> 211179 T669 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
>    [junit4]   2> 211179 T669 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>    [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>    [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>    [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>    [junit4]   2> 211184 T669 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 211184 T669 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 211185 T669 oasc.CoreContainer.load Host Name:
>    [junit4]   2> 211188 T670 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
>    [junit4]   2> 211188 T670 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
>    [junit4]   2> 211189 T670 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
>    [junit4]   2> 211189 T670 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
>    [junit4]   2> 211189 T670 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
>    [junit4]   2> 211227 T670 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>    [junit4]   2> 211265 T670 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
>    [junit4]   2> 211266 T670 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
>    [junit4]   2> 211270 T670 oass.IndexSchema.readSchema [collection1] Schema name=minimal
>    [junit4]   2> 211278 T670 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
>    [junit4]   2> 211279 T670 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
>    [junit4]   2> 211285 T670 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/
>    [junit4]   2> 211286 T670 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
>    [junit4]   2> 211286 T670 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238
>    [junit4]   2> 211286 T670 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index/
>    [junit4]   2> 211286 T670 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index' doesn't exist. Creating new index...
>    [junit4]   2> 211287 T670 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index
>    [junit4]   2> 211340 T670 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_1,generation=1}
>    [junit4]   2> 211340 T670 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>    [junit4]   2> 211341 T670 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 211342 T670 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>    [junit4]   2> 211342 T670 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>    [junit4]   2> 211344 T670 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>    [junit4]   2> 211346 T670 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>    [junit4]   2> 211347 T670 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>    [junit4]   2> 211347 T670 oass.SolrIndexSearcher.<init> Opening Searcher@f51a17[collection1] main
>    [junit4]   2> 211348 T671 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f51a17[collection1] main{StandardDirectoryReader(segments_1:1)}
>    [junit4]   2> 211348 T670 oasc.CoreContainer.registerCore registering core: collection1
>    [junit4]   2> 211350 T669 oas.SolrTestCaseJ4.initCore ####initCore end
>    [junit4]   2> ASYNC  NEW_CORE C142 name=collection1 org.apache.solr.core.SolrCore@1e7df7d
>    [junit4]   2> 211351 T669 C142 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_1,generation=1}
>    [junit4]   2> 211352 T669 C142 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>    [junit4]   2> 211353 T669 C142 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 3
>    [junit4]   2> 211355 T669 C142 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 211369 T669 C142 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_1,generation=1}
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_2,generation=2}
>    [junit4]   2> 211369 T669 C142 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
>    [junit4]   2> 211373 T669 C142 oass.SolrIndexSearcher.<init> Opening Searcher@105bc17[collection1] main
>    [junit4]   2> 211373 T669 C142 oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 211373 T671 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@105bc17[collection1] main{StandardDirectoryReader(segments_2:3 _0(5.0):c1)}
>    [junit4]   2> 211373 T669 C142 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 18
>    [junit4]   2> 211375 T669 oas.SolrTestCaseJ4.deleteCore ###deleteCore
>    [junit4]   2> 211375 T669 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=19970511
>    [junit4]   2> 211376 T669 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1e7df7d
>    [junit4]   2> 211376 T669 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
>    [junit4]   2> 211376 T669 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
>    [junit4]   2> 211376 T669 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
>    [junit4]   2> 211376 T669 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
>    [junit4]   2> 211377 T669 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
>    [junit4]   2> 211377 T669 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
>    [junit4]   2> 211377 T669 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238;done=false>>]
>    [junit4]   2> 211378 T669 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238
>    [junit4]   2> 211378 T669 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index;done=false>>]
>    [junit4]   2> 211378 T669 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index
>    [junit4]   2> NOTE: per solr.test.leavedatadir, dataDir will not be removed: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238
>    [junit4]   2> 211379 T669 oas.SolrTestCaseJ4.initCore ####initCore
>    [junit4]   2> 211379 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
>    [junit4]   2> 211380 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
>    [junit4]   2> 211380 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
>    [junit4]   2> 211380 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
>    [junit4]   2> 211422 T669 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>    [junit4]   2> 211460 T669 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
>    [junit4]   2> 211461 T669 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
>    [junit4]   2> 211465 T669 oass.IndexSchema.readSchema [null] Schema name=minimal
>    [junit4]   2> 211474 T669 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
>    [junit4]   2> 211475 T669 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>    [junit4]   2> 211475 T669 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
>    [junit4]   2> 211476 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
>    [junit4]   2> 211527 T669 oasc.CoreContainer.<init> New CoreContainer 24883180
>    [junit4]   2> 211528 T669 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
>    [junit4]   2> 211529 T669 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
>    [junit4]   2> 211529 T669 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
>    [junit4]   2> 211529 T669 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
>    [junit4]   2> 211529 T669 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>    [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>    [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>    [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>    [junit4]   2> 211534 T669 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 211534 T669 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 211534 T669 oasc.CoreContainer.load Host Name:
>    [junit4]   2> 211538 T673 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
>    [junit4]   2> 211538 T673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
>    [junit4]   2> 211538 T673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
>    [junit4]   2> 211539 T673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
>    [junit4]   2> 211539 T673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
>    [junit4]   2> 211573 T673 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>    [junit4]   2> 211614 T673 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
>    [junit4]   2> 211614 T673 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
>    [junit4]   2> 211618 T673 oass.IndexSchema.readSchema [collection1] Schema name=minimal
>    [junit4]   2> 211628 T673 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
>    [junit4]   2> 211629 T673 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
>    [junit4]   2> 211629 T673 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/
>    [junit4]   2> 211629 T673 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
>    [junit4]   2> 211630 T673 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238
>    [junit4]   2> 211630 T673 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index/
>    [junit4]   2> 211631 T673 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 211631 T673 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>    [junit4]   2> 211631 T673 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>    [junit4]   2> 211633 T673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>    [junit4]   2> 211635 T673 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>    [junit4]   2> 211636 T673 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>    [junit4]   2> 211636 T673 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index
>    [junit4]   2> 211637 T673 oass.SolrIndexSearcher.<init> Opening Searcher@af3192[collection1] main
>    [junit4]   2> 211637 T673 oasc.CoreContainer.registerCore registering core: collection1
>    [junit4]   2> 211638 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@af3192[collection1] main{StandardDirectoryReader(segments_2:3 _0(5.0):c1)}
>    [junit4]   2> 211640 T669 oas.SolrTestCaseJ4.initCore ####initCore end
>    [junit4]   2> 211645 T669 oas.SolrTestCaseJ4.setUp ###Starting testSharedCores
>    [junit4]   2> ASYNC  NEW_CORE C143 name=collection1 org.apache.solr.core.SolrCore@1ce0725
>    [junit4]   2> 211646 T669 C143 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
>    [junit4]   2> 211649 T669 C143 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_2,generation=2}
>    [junit4]   2> 211650 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
>    [junit4]   2> 211651 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 6
>    [junit4]   2> 211653 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
>    [junit4]   2> 211654 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 211667 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_2,generation=2}
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_3,generation=3}
>    [junit4]   2> 211668 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
>    [junit4]   2> 211670 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@3f22a5[collection1] main
>    [junit4]   2> 211670 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 211670 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f22a5[collection1] main{StandardDirectoryReader(segments_3:6 _1(5.0):c1)}
>    [junit4]   2> 211671 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 17
>    [junit4]   2> 211673 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 211707 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_3,generation=3}
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_4,generation=4}
>    [junit4]   2> 211708 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
>    [junit4]   2> 211712 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@17c32dc[collection1] main
>    [junit4]   2> 211712 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 211712 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17c32dc[collection1] main{StandardDirectoryReader(segments_4:8 _2(5.0):c1)}
>    [junit4]   2> 211713 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {optimize=} 0 40
>    [junit4]   2> 211717 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
>    [junit4]   2> 211719 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 211749 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_4,generation=4}
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_5,generation=5}
>    [junit4]   2> 211749 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
>    [junit4]   2> 211751 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@c1f4de[collection1] main
>    [junit4]   2> 211752 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 211752 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c1f4de[collection1] main{StandardDirectoryReader(segments_5:10 _2(5.0):c1 _3(5.0):c1)}
>    [junit4]   2> 211752 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 33
>    [junit4]   2> 211755 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
>    [junit4]   2> 211756 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 0
>    [junit4]   2> 211757 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 211777 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_5,generation=5}
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_6,generation=6}
>    [junit4]   2> 211778 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
>    [junit4]   2> 211782 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@a77af0[collection1] main
>    [junit4]   2> 211782 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 211782 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a77af0[collection1] main{StandardDirectoryReader(segments_6:12 _2(5.0):c1 _3(5.0):c1 _4(5.0):c2)}
>    [junit4]   2> 211783 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 26
>    [junit4]   2> 211785 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=foo2:doc} 0 0
>    [junit4]   2> 211786 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 211807 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_6,generation=6}
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_7,generation=7}
>    [junit4]   2> 211808 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
>    [junit4]   2> 211810 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@fdcabf[collection1] main
>    [junit4]   2> 211811 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 211811 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fdcabf[collection1] main{StandardDirectoryReader(segments_7:13 _2(5.0):c1 _3(5.0):c1 _4(5.0):c2/1:delGen=1)}
>    [junit4]   2> 211811 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 25
>    [junit4]   2> 211813 T669 oas.SolrTestCaseJ4.tearDown ###Ending testSharedCores
>    [junit4]   2> 211817 T669 oas.SolrTestCaseJ4.setUp ###Starting testReaderIsNotNRT
>    [junit4]   2> 211817 T669 C143 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
>    [junit4]   2> 211817 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 0
>    [junit4]   2> 211819 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
>    [junit4]   2> 211820 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 211839 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_7,generation=7}
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_8,generation=8}
>    [junit4]   2> 211839 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 8
>    [junit4]   2> 211845 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@497861[collection1] main
>    [junit4]   2> 211846 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 211846 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@497861[collection1] main{StandardDirectoryReader(segments_8:16 _5(5.0):c1)}
>    [junit4]   2> 211848 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 28
>    [junit4]   2> 211851 T669 oasc.CoreContainer.reload Reloading SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
>    [junit4]   2> 211852 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
>    [junit4]   2> 211852 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
>    [junit4]   2> 211852 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
>    [junit4]   2> 211853 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
>    [junit4]   2> 211895 T669 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>    [junit4]   2> 211939 T669 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
>    [junit4]   2> 211940 T669 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
>    [junit4]   2> 211944 T669 oass.IndexSchema.readSchema [collection1] Schema name=minimal
>    [junit4]   2> 211953 T669 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
>    [junit4]   2> 211955 T669 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/
>    [junit4]   2> 211955 T669 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
>    [junit4]   2> 211955 T669 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index/
>    [junit4]   2> 211956 T669 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 211957 T669 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>    [junit4]   2> 211957 T669 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>    [junit4]   2> 211959 T669 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>    [junit4]   2> 211961 T669 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>    [junit4]   2> 211962 T669 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>    [junit4]   2> 211963 T669 oass.SolrIndexSearcher.<init> Opening Searcher@925ff4[collection1] main
>    [junit4]   2> 211964 T669 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
>    [junit4]   2> 211964 T669 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
>    [junit4]   2> 211964 T669 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
>    [junit4]   2> 211964 T676 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@925ff4[collection1] main{StandardDirectoryReader(segments_8:16 _5(5.0):c1)}
>    [junit4]   2> 211966 T669 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_8,generation=8}
>    [junit4]   2> 211966 T669 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 8
>    [junit4]   2> 211966 T669 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
>    [junit4]   2> 211967 T669 oass.SolrIndexSearcher.<init> Opening Searcher@4effa[collection1] main
>    [junit4]   2> 211967 T676 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4effa[collection1] main{StandardDirectoryReader(segments_8:16 _5(5.0):c1)}
>    [junit4]   2> 211968 T669 oasc.CoreContainer.registerCore replacing core: collection1
>    [junit4]   2> 211968 T669 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1ce0725
>    [junit4]   2> 211969 T669 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=5,autocommits=0,soft autocommits=0,optimizes=1,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=3,cumulative_errors=0}
>    [junit4]   2> 211969 T669 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
>    [junit4]   2> ASYNC  NEW_CORE C144 name=collection1 org.apache.solr.core.SolrCore@270e4d
>    [junit4]   2> 211970 T669 C144 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
>    [junit4]   2> 211972 T669 C144 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 212039 T669 C144 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_8,generation=8}
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_9,generation=9}
>    [junit4]   2> 212040 T669 C144 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 9
>    [junit4]   2> 212045 T669 C144 oass.SolrIndexSearcher.<init> Opening Searcher@1ea1441[collection1] main
>    [junit4]   2> 212046 T669 C144 oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 212046 T676 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ea1441[collection1] main{StandardDirectoryReader(segments_9:18 _5(5.0):c1 _6(5.0):c1)}
>    [junit4]   2> 212047 T669 C144 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 75
>    [junit4]   2> 212053 T669 C144 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
>    [junit4]   2> 212056 T669 oasc.CoreContainer.reload Reloading SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
>    [junit4]   2> 212057 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
>    [junit4]   2> 212058 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
>    [junit4]   2> 212058 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
>    [junit4]   2> 212059 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
>    [junit4]   2> 212102 T669 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
>    [junit4]   2> 212150 T669 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
>    [junit4]   2> 212150 T669 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
>    [junit4]   2> 212155 T669 oass.IndexSchema.readSchema [collection1] Schema name=minimal
>    [junit4]   2> 212168 T669 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
>    [junit4]   2> 212170 T669 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/
>    [junit4]   2> 212170 T669 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
>    [junit4]   2> 212170 T669 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index/
>    [junit4]   2> 212172 T669 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 212172 T669 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>    [junit4]   2> 212173 T669 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>    [junit4]   2> 212176 T669 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>    [junit4]   2> 212179 T669 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>    [junit4]   2> 212179 T669 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>    [junit4]   2> 212181 T669 oass.SolrIndexSearcher.<init> Opening Searcher@6f1dd8[collection1] main
>    [junit4]   2> 212181 T669 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
>    [junit4]   2> 212182 T669 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
>    [junit4]   2> 212182 T669 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
>    [junit4]   2> 212182 T677 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6f1dd8[collection1] main{StandardDirectoryReader(segments_9:18 _5(5.0):c1 _6(5.0):c1)}
>    [junit4]   2> 212207 T669 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_9,generation=9}
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_a,generation=10}
>    [junit4]   2> 212208 T669 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 10
>    [junit4]   2> 212212 T669 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>                commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_a,generation=10}
>    [junit4]   2> 212212 T669 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 10
>    [junit4]   2> 212213 T669 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
>    [junit4]   2> 212215 T669 oass.SolrIndexSearcher.<init> Opening Searcher@b5c786[collection1] main
>    [junit4]   2> 212216 T669 oasc.CoreContainer.registerCore replacing core: collection1
>    [junit4]   2> 212216 T669 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@270e4d
>    [junit4]   2> 212216 T669 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
>    [junit4]   2> 212217 T669 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
>    [junit4]   2> 212217 T669 oas.SolrTestCaseJ4.tearDown ###Ending testReaderIsNotNRT
>    [junit4]   2> 212217 T677 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b5c786[collection1] main{StandardDirectoryReader(segments_a:20 _5(5.0):c1 _6(5.0):c1 _7(5.0):c1)}
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestNonNRTOpen -Dtests.method=testReaderIsNotNRT -Dtests.seed=F42EFCF46D93B00A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_HN -Dtests.timezone=Asia/Ashgabat -Dtests.file.encoding=UTF-8
>    [junit4] FAILURE 0.43s J0 | TestNonNRTOpen.testReaderIsNotNRT <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError: expected:<3> but was:<2>
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([F42EFCF46D93B00A:41A89D73D25202FE]:0)
>    [junit4]    >        at org.apache.solr.core.TestNonNRTOpen.assertNotNRT(TestNonNRTOpen.java:133)
>    [junit4]    >        at org.apache.solr.core.TestNonNRTOpen.testReaderIsNotNRT(TestNonNRTOpen.java:94)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:744)
>    [junit4]   2> 212240 T669 oas.SolrTestCaseJ4.deleteCore ###deleteCore
>    [junit4]   2> 212240 T669 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=24883180
>    [junit4]   2> 212240 T669 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3e32e1
>    [junit4]   2> 212240 T669 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
>    [junit4]   2> 212241 T669 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
>    [junit4]   2> 212241 T669 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
>    [junit4]   2> 212241 T669 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
>    [junit4]   2> 212241 T669 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
>    [junit4]   2> 212242 T669 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
>    [junit4]   2> 212242 T669 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index;done=false>>]
>    [junit4]   2> 212242 T669 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index
>    [junit4]   2> 212242 T669 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238;done=false>>]
>    [junit4]   2> 212243 T669 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238
>    [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=es_HN, timezone=Asia/Ashgabat
>    [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=54595472,total=200085504
>    [junit4]   2> NOTE: All tests run in this JVM: [SliceStateUpdateTest, TestAddFieldRealTimeGet, LegacyHTMLStripCharFilterTest, CircularListTest, ResponseLogComponentTest, TestSystemIdResolver, EchoParamsTest, TestSurroundQueryParser, TestSolrXmlPersistor, TestCoreContainer, DirectUpdateHandlerOptimizeTest, TestFreeTextSuggestions, PrimitiveFieldTypeTest, SpellPossibilityIteratorTest, NoFacetTest, ResourceLoaderTest, UnloadDistributedZkTest, SolrXmlInZkTest, TestBM25SimilarityFactory, TestUniqueKeyFieldResource, FieldMutatingUpdateProcessorTest, LukeRequestHandlerTest, TestRandomMergePolicy, TestElisionMultitermQuery, LeaderElectionTest, TestCollapseQParserPlugin, DocumentBuilderTest, QueryFacetTest, TestNonNRTOpen]
>    [junit4] Completed on J0 in 1.23s, 2 tests, 1 failure <<< FAILURES!
>
> [...truncated 450 lines...]
>    [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
>    [junit4]   2> 894100 T2509 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
>    [junit4]   2> 894101 T2509 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_vro/u
>    [junit4]   2> 894104 T2509 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
>    [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1393330517296
>    [junit4]   2> 894105 T2509 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>    [junit4]   2> 894105 T2510 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>    [junit4]   2> 894205 T2509 oasc.ZkTestServer.run start zk server on port:45107
>    [junit4]   2> 894207 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 894264 T2516 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ec6f75 name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 894264 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 894264 T2509 oascc.SolrZkClient.makePath makePath: /solr
>    [junit4]   2> 894269 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 894270 T2518 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14cbbe5 name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 894270 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 894270 T2509 oascc.SolrZkClient.makePath makePath: /collections/collection1
>    [junit4]   2> 894276 T2509 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>    [junit4]   2> 894279 T2509 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>    [junit4]   2> 894282 T2509 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>    [junit4]   2> 894285 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
>    [junit4]   2> 894285 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>    [junit4]   2> 894288 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
>    [junit4]   2> 894288 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>    [junit4]   2> 894291 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 894292 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 894294 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>    [junit4]   2> 894294 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>    [junit4]   2> 894296 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>    [junit4]   2> 894297 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>    [junit4]   2> 894298 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>    [junit4]   2> 894299 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>    [junit4]   2> 894301 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>    [junit4]   2> 894301 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>    [junit4]   2> 894303 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 894304 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 894305 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 894306 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 894308 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>    [junit4]   2> 894309 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>    [junit4]   2> 894311 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>    [junit4]   2> 894311 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>    [junit4]   2> 894315 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 894316 T2520 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1333d43 name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 894316 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> Using new style solr.xml
>    [junit4]   2> 894318 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 894319 T2522 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e3bfec name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 894319 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 894319 T2509 oascc.SolrZkClient.makePath makePath: /solr
>    [junit4]   2> 894322 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 894322 T2524 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@177a78d name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 894323 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 894323 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
>    [junit4]   2> 894324 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
>    [junit4]   2> 894327 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
>    [junit4]   2> 894327 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
>    [junit4]   2> 894330 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 894330 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 894332 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
>    [junit4]   2> 894332 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
>    [junit4]   2> 894334 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
>    [junit4]   2> 894334 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
>    [junit4]   2> 894337 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
>    [junit4]   2> 894337 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
>    [junit4]   2> 894339 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
>    [junit4]   2> 894339 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
>    [junit4]   2> 894341 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 894341 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 894344 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
>    [junit4]   2> 894344 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
>    [junit4]   2> 894346 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
>    [junit4]   2> 894346 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
>    [junit4]   2> 894348 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
>    [junit4]   2> 894349 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
>    [junit4]   2> 894582 T2509 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 894585 T2509 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>    [junit4]   2> 894588 T2509 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:40997
>    [junit4]   2> 894589 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>    [junit4]   2> 894590 T2509 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>    [junit4]   2> 894590 T2509 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542
>    [junit4]   2> 894590 T2509 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542/'
>    [junit4]   2> 894602 T2509 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542/solr.xml
>    [junit4]   2> 894660 T2509 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542
>    [junit4]   2> 894661 T2509 oasc.CoreContainer.<init> New CoreContainer 16971642
>    [junit4]   2> 894661 T2509 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542/]
>    [junit4]   2> 894662 T2509 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>    [junit4]   2> 894662 T2509 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
>    [junit4]   2> 894663 T2509 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>    [junit4]   2> 894663 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 894663 T2509 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>    [junit4]   2> 894664 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 894664 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>    [junit4]   2> 894664 T2509 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>    [junit4]   2> 894665 T2509 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>    [junit4]   2> 894668 T2509 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 894668 T2509 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 894668 T2509 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 894669 T2509 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45107/solr
>    [junit4]   2> 894669 T2509 oasc.ZkController.checkChrootPath zkHost includes chroot
>    [junit4]   2> 894670 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 894672 T2535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11ae851 name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 894672 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 894675 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 894675 T2537 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@176a7de name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 894676 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 894678 T2509 oascc.SolrZkClient.makePath makePath: /overseer/queue
>    [junit4]   2> 894682 T2509 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>    [junit4]   2> 894686 T2509 oascc.SolrZkClient.makePath makePath: /live_nodes
>    [junit4]   2> 894687 T2509 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40997__vro%2Fu
>    [junit4]   2> 894688 T2509 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40997__vro%2Fu
>    [junit4]   2> 894690 T2509 oascc.SolrZkClient.makePath makePath: /overseer_elect
>    [junit4]   2> 894691 T2509 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>    [junit4]   2> 894694 T2509 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40997__vro%2Fu
>    [junit4]   2> 894694 T2509 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>    [junit4]   2> 894696 T2509 oasc.Overseer.start Overseer (id=91313308781641734-127.0.0.1:40997__vro%2Fu-n_0000000000) starting
>    [junit4]   2> 894698 T2509 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>    [junit4]   2> 894701 T2539 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>    [junit4]   2> 894701 T2509 oascc.SolrZkClient.makePath makePath: /clusterstate.json
>    [junit4]   2> 894701 T2539 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
>    [junit4]   2> 894704 T2509 oascc.SolrZkClient.makePath makePath: /aliases.json
>    [junit4]   2> 894706 T2509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>    [junit4]   2> 894709 T2538 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>    [junit4]   2> 894711 T2509 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542
>    [junit4]   2> 894718 T2509 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542/conf/
>    [junit4]   2> 894825 T2509 oasc.CorePropertiesLocator.discover Found 1 core definitions
>    [junit4]   2> 894826 T2509 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
>    [junit4]   2> 894826 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>    [junit4]   2> 894829 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 894830 T2542 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c10eb name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 894830 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 894832 T2509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>    [junit4]   2> 894835 T2509 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
>    [junit4]   2> 895053 T2509 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
>    [junit4]   2> 895054 T2509 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 895056 T2509 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>    [junit4]   2> 895057 T2509 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42849
>    [junit4]   2> 895059 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>    [junit4]   2> 895059 T2509 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>    [junit4]   2> 895060 T2509 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027
>    [junit4]   2> 895060 T2509 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027/'
>    [junit4]   2> 895071 T2509 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027/solr.xml
>    [junit4]   2> 895112 T2509 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027
>    [junit4]   2> 895112 T2509 oasc.CoreContainer.<init> New CoreContainer 20771280
>    [junit4]   2> 895113 T2509 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027/]
>    [junit4]   2> 895114 T2509 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>    [junit4]   2> 895114 T2509 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
>    [junit4]   2> 895114 T2509 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>    [junit4]   2> 895115 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 895115 T2509 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>    [junit4]   2> 895115 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 895116 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>    [junit4]   2> 895116 T2509 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>    [junit4]   2> 895116 T2509 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>    [junit4]   2> 895120 T2509 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 895120 T2509 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 895120 T2509 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 895121 T2509 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45107/solr
>    [junit4]   2> 895121 T2509 oasc.ZkController.checkChrootPath zkHost includes chroot
>    [junit4]   2> 895122 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 895123 T2553 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e210e5 name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 895123 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 895125 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 895126 T2555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56fee8 name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 895127 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 895130 T2509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>    [junit4]   2> 896133 T2509 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42849__vro%2Fu
>    [junit4]   2> 896135 T2509 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42849__vro%2Fu
>    [junit4]   2> 896139 T2542 oascc.ZkStateReader$3.process Updating live nodes... (2)
>    [junit4]   2> 896139 T2537 oascc.ZkStateReader$3.process Updating live nodes... (2)
>    [junit4]   2> 896139 T2555 oascc.ZkStateReader$3.process Updating live nodes... (2)
>    [junit4]   2> 896146 T2509 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027
>    [junit4]   2> 896151 T2509 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027/conf/
>    [junit4]   2> 896243 T2509 oasc.CorePropertiesLocator.discover Found 1 core definitions
>    [junit4]   2> 896244 T2509 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
>    [junit4]   2> 896244 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>    [junit4]   2> 896463 T2509 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
>    [junit4]   2> 896464 T2509 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 896466 T2509 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>    [junit4]   2> 896467 T2509 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51548
>    [junit4]   2> 896469 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>    [junit4]   2> 896469 T2509 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>    [junit4]   2> 896469 T2509 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437
>    [junit4]   2> 896470 T2509 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437/'
>    [junit4]   2> 896481 T2509 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437/solr.xml
>    [junit4]   2> 896522 T2509 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437
>    [junit4]   2> 896523 T2509 oasc.CoreContainer.<init> New CoreContainer 15529496
>    [junit4]   2> 896523 T2509 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437/]
>    [junit4]   2> 896524 T2509 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>    [junit4]   2> 896525 T2509 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
>    [junit4]   2> 896525 T2509 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>    [junit4]   2> 896525 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 896526 T2509 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>    [junit4]   2> 896526 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 896526 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>    [junit4]   2> 896527 T2509 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>    [junit4]   2> 896527 T2509 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>    [junit4]   2> 896530 T2509 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 896531 T2509 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 896531 T2509 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 896531 T2509 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45107/solr
>    [junit4]   2> 896532 T2509 oasc.ZkController.checkChrootPath zkHost includes chroot
>    [junit4]   2> 896532 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 896534 T2567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1236d44 name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 896534 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 896537 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>    [junit4]   2> 896538 T2569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e436a2 name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 896539 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>    [junit4]   2> 896543 T2509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>    [junit4]   2> 897545 T2509 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51548__vro%2Fu
>    [junit4]   2> 897546 T2509 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51548__vro%2Fu
>    [junit4]   2> 897549 T2542 oascc.ZkStateReader$3.process Updating live nodes... (3)
>    [junit4]   2> 897549 T2555 oascc.ZkStateReader$3.process Updating live nodes... (3)
>    [junit4]   2> 897549 T2537 oascc.ZkStateReader$3.process Updating live nodes... (3)
>    [junit4]   2> 897549 T2569 oascc.ZkStateReader$3.process Updating live nodes... (3)
>    [junit4]   2> 897562 T2509 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437
>    [junit4]   2> 897569 T2509 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437/conf/
>    [junit4]   2> 897679 T2509 oasc.CorePropertiesLocator.discover Found 1 core definitions
>    [junit4]   2> 897680 T2509 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
>    [junit4]   2> 897680 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>    [junit4]   2> 897919 T2509 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
>    [junit4]   2> 897920 T2509 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 897922 T2509 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>    [junit4]   2> 897923 T2509 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:59103
>    [junit4]   2> 897925 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>    [junit4]   2> 897925 T2509 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>    [junit4]   2> 897926 T2509 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873
>    [junit4]   2> 897926 T2509 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873/'
>    [junit4]   2> 897937 T2509 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873/solr.xml
>    [junit4]   2> 897992 T2509 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873
>    [junit4]   2> 897993 T2509 oasc.CoreContainer.<init> New CoreContainer 24640474
>    [junit4]   2> 897993 T2509 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873/]
>    [junit4]   2> 897994 T2509 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>    [junit4]   2> 897994 T2509 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
>    [junit4]   2> 897995 T2509 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>    [junit4]   2> 897995 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 897995 T2509 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>    [junit4]   2> 897996 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 897996 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>    [junit4]   2> 897996 T2509 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>    [junit4]   2> 897997 T2509 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>    [junit4]   2> 898000 T2509 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 898000 T2509 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
>
> [...truncated too long message...]
>
> akePath: /collections/awholynewcollection_1/leaders/shard2
>    [junit4]   2> 1044660 T2555 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
>    [junit4]   2>                at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>    [junit4]   2>                at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>    [junit4]   2>                at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
>    [junit4]   2>                at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
>    [junit4]   2>                at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:203)
>    [junit4]   2>                at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>    [junit4]   2>                at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:203)
>    [junit4]   2>                at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:414)
>    [junit4]   2>                at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
>    [junit4]   2>                at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
>    [junit4]   2>                at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
>    [junit4]   2>                at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
>    [junit4]   2>                at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>    [junit4]   2>                at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>    [junit4]   2>                at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>    [junit4]   2>                at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>    [junit4]   2>                at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>    [junit4]   2>                at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>    [junit4]   2>
>    [junit4]   2> 1044661 T2555 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_1/leader_elect/shard2/election/91313308781641737-core_node5-n_0000000001
>    [junit4]   2>        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>    [junit4]   2>        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>    [junit4]   2>        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
>    [junit4]   2>        at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
>    [junit4]   2>        at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
>    [junit4]   2>        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>    [junit4]   2>        at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
>    [junit4]   2>        at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
>    [junit4]   2>        at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
>    [junit4]   2>        at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>    [junit4]   2>        at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>    [junit4]   2>        at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>    [junit4]   2>        at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>    [junit4]   2>        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>    [junit4]   2>        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>    [junit4]   2>
>    [junit4]   2> 1044661 T2555 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/nodes_used_collection/leader_elect/shard1/election
>    [junit4]   2>        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>    [junit4]   2>        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>    [junit4]   2>        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>    [junit4]   2>        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
>    [junit4]   2>        at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
>    [junit4]   2>        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>    [junit4]   2>        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
>    [junit4]   2>        at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>    [junit4]   2>        at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>    [junit4]   2>        at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>    [junit4]   2>        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>    [junit4]   2>        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>    [junit4]   2>
>    [junit4]   2> 1044662 T2555 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>    [junit4]   2> 1044662 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
>    [junit4]   2> 1044662 T2555 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>    [junit4]   2> 1044662 T2555 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1044663 T2555 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1044663 T2555 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=fr_CA, timezone=SystemV/YST9YDT
>    [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=157898200,total=391720960
>    [junit4]   2> NOTE: All tests run in this JVM: [TestFieldTypeCollectionResource, HighlighterConfigTest, TestImplicitCoreProperties, TestCSVResponseWriter, DirectUpdateHandlerTest, FullSolrCloudDistribCmdsTest, DebugComponentTest, ChaosMonkeySafeLeaderTest, DistributedQueryComponentOptimizationTest, TestReloadAndDeleteDocs, TestStandardQParsers, TestRecovery, TestRTGBase, TermVectorComponentDistributedTest, TestIndexSearcher, DateFieldTest, TestFieldTypeResource, TestFastOutputStream, NumericFieldsTest, TestWordDelimiterFilterFactory, DistanceFunctionTest, TestNonDefinedSimilarityFactory, TestCursorMarkWithoutUniqueKey, DistributedQueryComponentCustomSortTest, TestStressLucene, EnumFieldTest, SolrTestCaseJ4Test, TestQuerySenderNoQuery, HdfsBasicDistributedZkTest, TestPartialUpdateDeduplication, TestBinaryResponseWriter, TestOmitPositions, TestQueryTypes, SolrCmdDistributorTest, TestDistribDocBasedVersion, SpellCheckComponentTest, DistribCursorPagingTest, TestDFRSimilarityFactory, TestNumberUtils, TestNoOpRegenerator, TestClassNameShortening, ConnectionManagerTest, HdfsBasicDistributedZk2Test, HdfsLockFactoryTest, FieldAnalysisRequestHandlerTest, PathHierarchyTokenizerFactoryTest, BasicZkTest, MultiTermTest, TestCollationField, TestIndexingPerformance, TestDistributedGrouping, SpellCheckCollatorTest, TestDynamicFieldCollectionResource, DirectSolrSpellCheckerTest, XmlUpdateRequestHandlerTest, PingRequestHandlerTest, TestComponentsName, HdfsCollectionsAPIDistributedZkTest, MinimalSchemaTest, DeleteReplicaTest, TimeZoneUtilsTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, HdfsChaosMonkeySafeLeaderTest, HdfsUnloadDistributedZkTest, TestSolrDeletionPolicy2, TestBinaryField, ZkNodePropsTest, AssignTest, TestRangeQuery, OpenExchangeRatesOrgProviderTest, FileUtilsTest, DocValuesMultiTest, AlternateDirectoryTest, TestUpdate, RAMDirectoryFactoryTest, PrimUtilsTest, JSONWriterTest, IndexBasedSpellCheckerTest, TestShardHandlerFactory, TestSerializedLuceneMatchVersion, ZkSolrClientTest, XsltUpdateRequestHandlerTest, ConvertedLegacyTest, OpenCloseCoreStressTest, SuggestComponentTest, TestReload, TestMaxScoreQueryParser, CollectionsAPIDistributedZkTest]
>    [junit4] Completed on J1 in 151.56s, 1 test, 1 failure <<< FAILURES!
>
> [...truncated 577 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1606 tests, 2 failures, 35 ignored (7 assumptions)
>
> Total time: 58 minutes 46 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseSerialGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org



-- 
Regards,
Shalin Shekhar Mangar.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9591 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9591/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.core.TestNonNRTOpen.testReaderIsNotNRT

Error Message:
expected:<3> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<3> but was:<2>
	at __randomizedtesting.SeedInfo.seed([F42EFCF46D93B00A:41A89D73D25202FE]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.core.TestNonNRTOpen.assertNotNRT(TestNonNRTOpen.java:133)
	at org.apache.solr.core.TestNonNRTOpen.testReaderIsNotNRT(TestNonNRTOpen.java:94)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Could not find collection:addReplicaColl

Stack Trace:
java.lang.AssertionError: Could not find collection:addReplicaColl
	at __randomizedtesting.SeedInfo.seed([F42EFCF46D93B00A:75C872EC1ACCD036]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:147)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:724)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1193)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10032 lines...]
   [junit4] Suite: org.apache.solr.core.TestNonNRTOpen
   [junit4]   2> 211034 T669 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 211035 T669 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238
   [junit4]   2> 211035 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 211036 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 211036 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 211036 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 211074 T669 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 211111 T669 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
   [junit4]   2> 211111 T669 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
   [junit4]   2> 211115 T669 oass.IndexSchema.readSchema [null] Schema name=minimal
   [junit4]   2> 211128 T669 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
   [junit4]   2> 211130 T669 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 211130 T669 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 211130 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 211177 T669 oasc.CoreContainer.<init> New CoreContainer 19970511
   [junit4]   2> 211178 T669 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 211179 T669 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 211179 T669 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 211179 T669 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 211179 T669 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 211180 T669 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 211184 T669 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 211184 T669 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 211185 T669 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 211188 T670 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 211188 T670 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 211189 T670 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 211189 T670 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 211189 T670 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 211227 T670 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 211265 T670 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
   [junit4]   2> 211266 T670 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
   [junit4]   2> 211270 T670 oass.IndexSchema.readSchema [collection1] Schema name=minimal
   [junit4]   2> 211278 T670 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
   [junit4]   2> 211279 T670 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 211285 T670 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/
   [junit4]   2> 211286 T670 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 211286 T670 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238
   [junit4]   2> 211286 T670 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index/
   [junit4]   2> 211286 T670 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index' doesn't exist. Creating new index...
   [junit4]   2> 211287 T670 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index
   [junit4]   2> 211340 T670 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_1,generation=1}
   [junit4]   2> 211340 T670 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 211341 T670 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 211342 T670 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 211342 T670 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 211344 T670 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 211346 T670 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 211347 T670 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 211347 T670 oass.SolrIndexSearcher.<init> Opening Searcher@f51a17[collection1] main
   [junit4]   2> 211348 T671 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f51a17[collection1] main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 211348 T670 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 211350 T669 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> ASYNC  NEW_CORE C142 name=collection1 org.apache.solr.core.SolrCore@1e7df7d
   [junit4]   2> 211351 T669 C142 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_1,generation=1}
   [junit4]   2> 211352 T669 C142 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 211353 T669 C142 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 3
   [junit4]   2> 211355 T669 C142 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 211369 T669 C142 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_2,generation=2}
   [junit4]   2> 211369 T669 C142 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 211373 T669 C142 oass.SolrIndexSearcher.<init> Opening Searcher@105bc17[collection1] main
   [junit4]   2> 211373 T669 C142 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 211373 T671 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@105bc17[collection1] main{StandardDirectoryReader(segments_2:3 _0(5.0):c1)}
   [junit4]   2> 211373 T669 C142 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 18
   [junit4]   2> 211375 T669 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 211375 T669 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=19970511
   [junit4]   2> 211376 T669 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1e7df7d
   [junit4]   2> 211376 T669 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 211376 T669 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 211376 T669 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 211376 T669 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 211377 T669 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 211377 T669 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 211377 T669 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238;done=false>>]
   [junit4]   2> 211378 T669 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238
   [junit4]   2> 211378 T669 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index;done=false>>]
   [junit4]   2> 211378 T669 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index
   [junit4]   2> NOTE: per solr.test.leavedatadir, dataDir will not be removed: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238
   [junit4]   2> 211379 T669 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 211379 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 211380 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 211380 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 211380 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 211422 T669 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 211460 T669 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
   [junit4]   2> 211461 T669 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
   [junit4]   2> 211465 T669 oass.IndexSchema.readSchema [null] Schema name=minimal
   [junit4]   2> 211474 T669 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
   [junit4]   2> 211475 T669 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 211475 T669 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 211476 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 211527 T669 oasc.CoreContainer.<init> New CoreContainer 24883180
   [junit4]   2> 211528 T669 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 211529 T669 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 211529 T669 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 211529 T669 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 211529 T669 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 211530 T669 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 211534 T669 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 211534 T669 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 211534 T669 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 211538 T673 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 211538 T673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 211538 T673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 211539 T673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 211539 T673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 211573 T673 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 211614 T673 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
   [junit4]   2> 211614 T673 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
   [junit4]   2> 211618 T673 oass.IndexSchema.readSchema [collection1] Schema name=minimal
   [junit4]   2> 211628 T673 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
   [junit4]   2> 211629 T673 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 211629 T673 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/
   [junit4]   2> 211629 T673 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 211630 T673 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238
   [junit4]   2> 211630 T673 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index/
   [junit4]   2> 211631 T673 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 211631 T673 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 211631 T673 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 211633 T673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 211635 T673 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 211636 T673 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 211636 T673 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index
   [junit4]   2> 211637 T673 oass.SolrIndexSearcher.<init> Opening Searcher@af3192[collection1] main
   [junit4]   2> 211637 T673 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 211638 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@af3192[collection1] main{StandardDirectoryReader(segments_2:3 _0(5.0):c1)}
   [junit4]   2> 211640 T669 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 211645 T669 oas.SolrTestCaseJ4.setUp ###Starting testSharedCores
   [junit4]   2> ASYNC  NEW_CORE C143 name=collection1 org.apache.solr.core.SolrCore@1ce0725
   [junit4]   2> 211646 T669 C143 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 211649 T669 C143 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_2,generation=2}
   [junit4]   2> 211650 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 211651 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 6
   [junit4]   2> 211653 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
   [junit4]   2> 211654 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 211667 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_3,generation=3}
   [junit4]   2> 211668 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 211670 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@3f22a5[collection1] main
   [junit4]   2> 211670 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 211670 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f22a5[collection1] main{StandardDirectoryReader(segments_3:6 _1(5.0):c1)}
   [junit4]   2> 211671 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 17
   [junit4]   2> 211673 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 211707 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_3,generation=3}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_4,generation=4}
   [junit4]   2> 211708 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
   [junit4]   2> 211712 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@17c32dc[collection1] main
   [junit4]   2> 211712 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 211712 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17c32dc[collection1] main{StandardDirectoryReader(segments_4:8 _2(5.0):c1)}
   [junit4]   2> 211713 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {optimize=} 0 40
   [junit4]   2> 211717 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
   [junit4]   2> 211719 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 211749 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_4,generation=4}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_5,generation=5}
   [junit4]   2> 211749 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
   [junit4]   2> 211751 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@c1f4de[collection1] main
   [junit4]   2> 211752 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 211752 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c1f4de[collection1] main{StandardDirectoryReader(segments_5:10 _2(5.0):c1 _3(5.0):c1)}
   [junit4]   2> 211752 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 33
   [junit4]   2> 211755 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
   [junit4]   2> 211756 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 0
   [junit4]   2> 211757 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 211777 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_5,generation=5}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_6,generation=6}
   [junit4]   2> 211778 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
   [junit4]   2> 211782 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@a77af0[collection1] main
   [junit4]   2> 211782 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 211782 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a77af0[collection1] main{StandardDirectoryReader(segments_6:12 _2(5.0):c1 _3(5.0):c1 _4(5.0):c2)}
   [junit4]   2> 211783 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 26
   [junit4]   2> 211785 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=foo2:doc} 0 0
   [junit4]   2> 211786 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 211807 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_6,generation=6}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_7,generation=7}
   [junit4]   2> 211808 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
   [junit4]   2> 211810 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@fdcabf[collection1] main
   [junit4]   2> 211811 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 211811 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fdcabf[collection1] main{StandardDirectoryReader(segments_7:13 _2(5.0):c1 _3(5.0):c1 _4(5.0):c2/1:delGen=1)}
   [junit4]   2> 211811 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 25
   [junit4]   2> 211813 T669 oas.SolrTestCaseJ4.tearDown ###Ending testSharedCores
   [junit4]   2> 211817 T669 oas.SolrTestCaseJ4.setUp ###Starting testReaderIsNotNRT
   [junit4]   2> 211817 T669 C143 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 211817 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 0
   [junit4]   2> 211819 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
   [junit4]   2> 211820 T669 C143 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 211839 T669 C143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_7,generation=7}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_8,generation=8}
   [junit4]   2> 211839 T669 C143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 8
   [junit4]   2> 211845 T669 C143 oass.SolrIndexSearcher.<init> Opening Searcher@497861[collection1] main
   [junit4]   2> 211846 T669 C143 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 211846 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@497861[collection1] main{StandardDirectoryReader(segments_8:16 _5(5.0):c1)}
   [junit4]   2> 211848 T669 C143 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 28
   [junit4]   2> 211851 T669 oasc.CoreContainer.reload Reloading SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 211852 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 211852 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 211852 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 211853 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 211895 T669 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 211939 T669 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
   [junit4]   2> 211940 T669 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
   [junit4]   2> 211944 T669 oass.IndexSchema.readSchema [collection1] Schema name=minimal
   [junit4]   2> 211953 T669 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
   [junit4]   2> 211955 T669 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/
   [junit4]   2> 211955 T669 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 211955 T669 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index/
   [junit4]   2> 211956 T669 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 211957 T669 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 211957 T669 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 211959 T669 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 211961 T669 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 211962 T669 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 211963 T669 oass.SolrIndexSearcher.<init> Opening Searcher@925ff4[collection1] main
   [junit4]   2> 211964 T669 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
   [junit4]   2> 211964 T669 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
   [junit4]   2> 211964 T669 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
   [junit4]   2> 211964 T676 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@925ff4[collection1] main{StandardDirectoryReader(segments_8:16 _5(5.0):c1)}
   [junit4]   2> 211966 T669 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_8,generation=8}
   [junit4]   2> 211966 T669 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 8
   [junit4]   2> 211966 T669 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
   [junit4]   2> 211967 T669 oass.SolrIndexSearcher.<init> Opening Searcher@4effa[collection1] main
   [junit4]   2> 211967 T676 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4effa[collection1] main{StandardDirectoryReader(segments_8:16 _5(5.0):c1)}
   [junit4]   2> 211968 T669 oasc.CoreContainer.registerCore replacing core: collection1
   [junit4]   2> 211968 T669 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1ce0725
   [junit4]   2> 211969 T669 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=5,autocommits=0,soft autocommits=0,optimizes=1,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=3,cumulative_errors=0}
   [junit4]   2> 211969 T669 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> ASYNC  NEW_CORE C144 name=collection1 org.apache.solr.core.SolrCore@270e4d
   [junit4]   2> 211970 T669 C144 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
   [junit4]   2> 211972 T669 C144 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 212039 T669 C144 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_8,generation=8}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_9,generation=9}
   [junit4]   2> 212040 T669 C144 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 9
   [junit4]   2> 212045 T669 C144 oass.SolrIndexSearcher.<init> Opening Searcher@1ea1441[collection1] main
   [junit4]   2> 212046 T669 C144 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 212046 T676 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ea1441[collection1] main{StandardDirectoryReader(segments_9:18 _5(5.0):c1 _6(5.0):c1)}
   [junit4]   2> 212047 T669 C144 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 75
   [junit4]   2> 212053 T669 C144 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {add=[(null)]} 0 1
   [junit4]   2> 212056 T669 oasc.CoreContainer.reload Reloading SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 212057 T669 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 212058 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 212058 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 212059 T669 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 212102 T669 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 212150 T669 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-basic.xml
   [junit4]   2> 212150 T669 oass.IndexSchema.readSchema Reading Solr Schema from schema-minimal.xml
   [junit4]   2> 212155 T669 oass.IndexSchema.readSchema [collection1] Schema name=minimal
   [junit4]   2> 212168 T669 oass.IndexSchema.readSchema WARN no uniqueKey specified in schema.
   [junit4]   2> 212170 T669 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/
   [junit4]   2> 212170 T669 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 212170 T669 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestNonNRTOpen-1393329834238/index/
   [junit4]   2> 212172 T669 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 212172 T669 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 212173 T669 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 212176 T669 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 212179 T669 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 212179 T669 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 212181 T669 oass.SolrIndexSearcher.<init> Opening Searcher@6f1dd8[collection1] main
   [junit4]   2> 212181 T669 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
   [junit4]   2> 212182 T669 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
   [junit4]   2> 212182 T669 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
   [junit4]   2> 212182 T677 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6f1dd8[collection1] main{StandardDirectoryReader(segments_9:18 _5(5.0):c1 _6(5.0):c1)}
   [junit4]   2> 212207 T669 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_9,generation=9}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_a,generation=10}
   [junit4]   2> 212208 T669 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 10
   [junit4]   2> 212212 T669 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index,segFN=segments_a,generation=10}
   [junit4]   2> 212212 T669 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 10
   [junit4]   2> 212213 T669 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
   [junit4]   2> 212215 T669 oass.SolrIndexSearcher.<init> Opening Searcher@b5c786[collection1] main
   [junit4]   2> 212216 T669 oasc.CoreContainer.registerCore replacing core: collection1
   [junit4]   2> 212216 T669 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@270e4d
   [junit4]   2> 212216 T669 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 212217 T669 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 212217 T669 oas.SolrTestCaseJ4.tearDown ###Ending testReaderIsNotNRT
   [junit4]   2> 212217 T677 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b5c786[collection1] main{StandardDirectoryReader(segments_a:20 _5(5.0):c1 _6(5.0):c1 _7(5.0):c1)}
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestNonNRTOpen -Dtests.method=testReaderIsNotNRT -Dtests.seed=F42EFCF46D93B00A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_HN -Dtests.timezone=Asia/Ashgabat -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.43s J0 | TestNonNRTOpen.testReaderIsNotNRT <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<3> but was:<2>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F42EFCF46D93B00A:41A89D73D25202FE]:0)
   [junit4]    > 	at org.apache.solr.core.TestNonNRTOpen.assertNotNRT(TestNonNRTOpen.java:133)
   [junit4]    > 	at org.apache.solr.core.TestNonNRTOpen.testReaderIsNotNRT(TestNonNRTOpen.java:94)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 212240 T669 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 212240 T669 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=24883180
   [junit4]   2> 212240 T669 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3e32e1
   [junit4]   2> 212240 T669 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 212241 T669 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 212241 T669 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 212241 T669 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 212241 T669 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 212242 T669 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 212242 T669 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index;done=false>>]
   [junit4]   2> 212242 T669 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238/index
   [junit4]   2> 212242 T669 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238;done=false>>]
   [junit4]   2> 212243 T669 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/solrtest-TestNonNRTOpen-1393329834238
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=es_HN, timezone=Asia/Ashgabat
   [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=54595472,total=200085504
   [junit4]   2> NOTE: All tests run in this JVM: [SliceStateUpdateTest, TestAddFieldRealTimeGet, LegacyHTMLStripCharFilterTest, CircularListTest, ResponseLogComponentTest, TestSystemIdResolver, EchoParamsTest, TestSurroundQueryParser, TestSolrXmlPersistor, TestCoreContainer, DirectUpdateHandlerOptimizeTest, TestFreeTextSuggestions, PrimitiveFieldTypeTest, SpellPossibilityIteratorTest, NoFacetTest, ResourceLoaderTest, UnloadDistributedZkTest, SolrXmlInZkTest, TestBM25SimilarityFactory, TestUniqueKeyFieldResource, FieldMutatingUpdateProcessorTest, LukeRequestHandlerTest, TestRandomMergePolicy, TestElisionMultitermQuery, LeaderElectionTest, TestCollapseQParserPlugin, DocumentBuilderTest, QueryFacetTest, TestNonNRTOpen]
   [junit4] Completed on J0 in 1.23s, 2 tests, 1 failure <<< FAILURES!

[...truncated 450 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 894100 T2509 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 894101 T2509 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_vro/u
   [junit4]   2> 894104 T2509 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1393330517296
   [junit4]   2> 894105 T2509 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 894105 T2510 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 894205 T2509 oasc.ZkTestServer.run start zk server on port:45107
   [junit4]   2> 894207 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 894264 T2516 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ec6f75 name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 894264 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 894264 T2509 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 894269 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 894270 T2518 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14cbbe5 name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 894270 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 894270 T2509 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 894276 T2509 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 894279 T2509 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 894282 T2509 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 894285 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 894285 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 894288 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 894288 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 894291 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 894292 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 894294 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 894294 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 894296 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 894297 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 894298 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 894299 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 894301 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 894301 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 894303 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 894304 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 894305 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 894306 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 894308 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 894309 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 894311 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 894311 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 894315 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 894316 T2520 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1333d43 name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 894316 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 894318 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 894319 T2522 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e3bfec name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 894319 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 894319 T2509 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 894322 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 894322 T2524 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@177a78d name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 894323 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 894323 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 894324 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 894327 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 894327 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 894330 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 894330 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 894332 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 894332 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 894334 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 894334 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 894337 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 894337 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 894339 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 894339 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 894341 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 894341 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 894344 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 894344 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 894346 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 894346 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 894348 T2509 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 894349 T2509 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 894582 T2509 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 894585 T2509 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 894588 T2509 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:40997
   [junit4]   2> 894589 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 894590 T2509 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 894590 T2509 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542
   [junit4]   2> 894590 T2509 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542/'
   [junit4]   2> 894602 T2509 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542/solr.xml
   [junit4]   2> 894660 T2509 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542
   [junit4]   2> 894661 T2509 oasc.CoreContainer.<init> New CoreContainer 16971642
   [junit4]   2> 894661 T2509 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542/]
   [junit4]   2> 894662 T2509 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 894662 T2509 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 894663 T2509 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 894663 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 894663 T2509 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 894664 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 894664 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 894664 T2509 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 894665 T2509 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 894668 T2509 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 894668 T2509 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 894668 T2509 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 894669 T2509 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45107/solr
   [junit4]   2> 894669 T2509 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 894670 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 894672 T2535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11ae851 name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 894672 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 894675 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 894675 T2537 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@176a7de name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 894676 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 894678 T2509 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 894682 T2509 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 894686 T2509 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 894687 T2509 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40997__vro%2Fu
   [junit4]   2> 894688 T2509 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40997__vro%2Fu
   [junit4]   2> 894690 T2509 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 894691 T2509 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 894694 T2509 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40997__vro%2Fu
   [junit4]   2> 894694 T2509 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 894696 T2509 oasc.Overseer.start Overseer (id=91313308781641734-127.0.0.1:40997__vro%2Fu-n_0000000000) starting
   [junit4]   2> 894698 T2509 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 894701 T2539 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 894701 T2509 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 894701 T2539 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 894704 T2509 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 894706 T2509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 894709 T2538 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 894711 T2509 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542
   [junit4]   2> 894718 T2509 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393330517542/conf/
   [junit4]   2> 894825 T2509 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 894826 T2509 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 894826 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 894829 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 894830 T2542 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c10eb name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 894830 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 894832 T2509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 894835 T2509 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 895053 T2509 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 895054 T2509 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 895056 T2509 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 895057 T2509 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42849
   [junit4]   2> 895059 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 895059 T2509 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 895060 T2509 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027
   [junit4]   2> 895060 T2509 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027/'
   [junit4]   2> 895071 T2509 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027/solr.xml
   [junit4]   2> 895112 T2509 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027
   [junit4]   2> 895112 T2509 oasc.CoreContainer.<init> New CoreContainer 20771280
   [junit4]   2> 895113 T2509 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027/]
   [junit4]   2> 895114 T2509 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 895114 T2509 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 895114 T2509 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 895115 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 895115 T2509 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 895115 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 895116 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 895116 T2509 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 895116 T2509 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 895120 T2509 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 895120 T2509 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 895120 T2509 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 895121 T2509 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45107/solr
   [junit4]   2> 895121 T2509 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 895122 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 895123 T2553 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e210e5 name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 895123 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 895125 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 895126 T2555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56fee8 name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 895127 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 895130 T2509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 896133 T2509 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42849__vro%2Fu
   [junit4]   2> 896135 T2509 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42849__vro%2Fu
   [junit4]   2> 896139 T2542 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 896139 T2537 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 896139 T2555 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 896146 T2509 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027
   [junit4]   2> 896151 T2509 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393330518027/conf/
   [junit4]   2> 896243 T2509 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 896244 T2509 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 896244 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 896463 T2509 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 896464 T2509 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 896466 T2509 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 896467 T2509 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51548
   [junit4]   2> 896469 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 896469 T2509 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 896469 T2509 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437
   [junit4]   2> 896470 T2509 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437/'
   [junit4]   2> 896481 T2509 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437/solr.xml
   [junit4]   2> 896522 T2509 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437
   [junit4]   2> 896523 T2509 oasc.CoreContainer.<init> New CoreContainer 15529496
   [junit4]   2> 896523 T2509 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437/]
   [junit4]   2> 896524 T2509 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 896525 T2509 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 896525 T2509 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 896525 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 896526 T2509 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 896526 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 896526 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 896527 T2509 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 896527 T2509 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 896530 T2509 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 896531 T2509 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 896531 T2509 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 896531 T2509 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45107/solr
   [junit4]   2> 896532 T2509 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 896532 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 896534 T2567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1236d44 name:ZooKeeperConnection Watcher:127.0.0.1:45107 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 896534 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 896537 T2509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 896538 T2569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e436a2 name:ZooKeeperConnection Watcher:127.0.0.1:45107/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 896539 T2509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 896543 T2509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 897545 T2509 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51548__vro%2Fu
   [junit4]   2> 897546 T2509 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51548__vro%2Fu
   [junit4]   2> 897549 T2542 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 897549 T2555 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 897549 T2537 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 897549 T2569 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 897562 T2509 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437
   [junit4]   2> 897569 T2509 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393330519437/conf/
   [junit4]   2> 897679 T2509 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 897680 T2509 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 897680 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 897919 T2509 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 897920 T2509 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 897922 T2509 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 897923 T2509 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:59103
   [junit4]   2> 897925 T2509 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 897925 T2509 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 897926 T2509 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873
   [junit4]   2> 897926 T2509 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873/'
   [junit4]   2> 897937 T2509 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873/solr.xml
   [junit4]   2> 897992 T2509 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873
   [junit4]   2> 897993 T2509 oasc.CoreContainer.<init> New CoreContainer 24640474
   [junit4]   2> 897993 T2509 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393330520873/]
   [junit4]   2> 897994 T2509 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 897994 T2509 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 897995 T2509 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 897995 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 897995 T2509 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 897996 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 897996 T2509 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 897996 T2509 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 897997 T2509 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 898000 T2509 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 898000 T2509 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener

[...truncated too long message...]

akePath: /collections/awholynewcollection_1/leaders/shard2
   [junit4]   2> 1044660 T2555 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:414)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 1044661 T2555 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_1/leader_elect/shard2/election/91313308781641737-core_node5-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1044661 T2555 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/nodes_used_collection/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1044662 T2555 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1044662 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1044662 T2555 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1044662 T2555 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1044663 T2555 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1044663 T2555 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=fr_CA, timezone=SystemV/YST9YDT
   [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=157898200,total=391720960
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldTypeCollectionResource, HighlighterConfigTest, TestImplicitCoreProperties, TestCSVResponseWriter, DirectUpdateHandlerTest, FullSolrCloudDistribCmdsTest, DebugComponentTest, ChaosMonkeySafeLeaderTest, DistributedQueryComponentOptimizationTest, TestReloadAndDeleteDocs, TestStandardQParsers, TestRecovery, TestRTGBase, TermVectorComponentDistributedTest, TestIndexSearcher, DateFieldTest, TestFieldTypeResource, TestFastOutputStream, NumericFieldsTest, TestWordDelimiterFilterFactory, DistanceFunctionTest, TestNonDefinedSimilarityFactory, TestCursorMarkWithoutUniqueKey, DistributedQueryComponentCustomSortTest, TestStressLucene, EnumFieldTest, SolrTestCaseJ4Test, TestQuerySenderNoQuery, HdfsBasicDistributedZkTest, TestPartialUpdateDeduplication, TestBinaryResponseWriter, TestOmitPositions, TestQueryTypes, SolrCmdDistributorTest, TestDistribDocBasedVersion, SpellCheckComponentTest, DistribCursorPagingTest, TestDFRSimilarityFactory, TestNumberUtils, TestNoOpRegenerator, TestClassNameShortening, ConnectionManagerTest, HdfsBasicDistributedZk2Test, HdfsLockFactoryTest, FieldAnalysisRequestHandlerTest, PathHierarchyTokenizerFactoryTest, BasicZkTest, MultiTermTest, TestCollationField, TestIndexingPerformance, TestDistributedGrouping, SpellCheckCollatorTest, TestDynamicFieldCollectionResource, DirectSolrSpellCheckerTest, XmlUpdateRequestHandlerTest, PingRequestHandlerTest, TestComponentsName, HdfsCollectionsAPIDistributedZkTest, MinimalSchemaTest, DeleteReplicaTest, TimeZoneUtilsTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, HdfsChaosMonkeySafeLeaderTest, HdfsUnloadDistributedZkTest, TestSolrDeletionPolicy2, TestBinaryField, ZkNodePropsTest, AssignTest, TestRangeQuery, OpenExchangeRatesOrgProviderTest, FileUtilsTest, DocValuesMultiTest, AlternateDirectoryTest, TestUpdate, RAMDirectoryFactoryTest, PrimUtilsTest, JSONWriterTest, IndexBasedSpellCheckerTest, TestShardHandlerFactory, TestSerializedLuceneMatchVersion, ZkSolrClientTest, XsltUpdateRequestHandlerTest, ConvertedLegacyTest, OpenCloseCoreStressTest, SuggestComponentTest, TestReload, TestMaxScoreQueryParser, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 151.56s, 1 test, 1 failure <<< FAILURES!

[...truncated 577 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1606 tests, 2 failures, 35 ignored (7 assumptions)

Total time: 58 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-fcs-b129) - Build # 9590 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9590/
Java: 64bit/jdk1.8.0-fcs-b129 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Could not find collection:addReplicaColl

Stack Trace:
java.lang.AssertionError: Could not find collection:addReplicaColl
	at __randomizedtesting.SeedInfo.seed([F4DAC13CB118574B:753C4F24C6473777]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:147)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:724)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1193)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10432 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 833935 T4793 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 833935 T4793 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /gyu/
   [junit4]   2> 833938 T4793 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1393323853912
   [junit4]   2> 833939 T4793 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 833939 T4794 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 834039 T4793 oasc.ZkTestServer.run start zk server on port:47779
   [junit4]   2> 834041 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 834176 T4800 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@96d36ec name:ZooKeeperConnection Watcher:127.0.0.1:47779 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 834177 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 834177 T4793 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 834183 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 834184 T4802 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25dd4414 name:ZooKeeperConnection Watcher:127.0.0.1:47779/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 834184 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 834185 T4793 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 834189 T4793 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 834192 T4793 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 834195 T4793 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 834198 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 834199 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 834205 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 834205 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 834210 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 834210 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 834215 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 834216 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 834221 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 834221 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 834223 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 834223 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 834225 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 834226 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 834230 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 834230 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 834232 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 834233 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 834235 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 834236 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 834239 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 834239 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 834242 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 834243 T4804 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d15b640 name:ZooKeeperConnection Watcher:127.0.0.1:47779 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 834243 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 834243 T4793 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 834245 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 834246 T4806 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6437b048 name:ZooKeeperConnection Watcher:127.0.0.1:47779/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 834246 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 834246 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 834247 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 834250 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 834250 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 834252 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 834252 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 834255 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 834255 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 834258 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 834258 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 834260 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 834260 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 834261 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 834262 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 834264 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 834265 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 834267 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 834267 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 834269 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 834270 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 834271 T4793 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 834272 T4793 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 834519 T4793 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 834524 T4793 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39617
   [junit4]   2> 834524 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 834524 T4793 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 834525 T4793 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248
   [junit4]   2> 834525 T4793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/'
   [junit4]   2> 834534 T4793 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solr.xml
   [junit4]   2> 834541 T4793 oasc.CoreContainer.<init> New CoreContainer 1596245000
   [junit4]   2> 834542 T4793 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/]
   [junit4]   2> 834542 T4793 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 834542 T4793 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 834542 T4793 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 834543 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 834543 T4793 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 834543 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 834543 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 834544 T4793 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 834544 T4793 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 834545 T4793 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 834546 T4793 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 834546 T4793 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 834546 T4793 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47779/solr
   [junit4]   2> 834547 T4793 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 834547 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 834548 T4817 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@634bdf9c name:ZooKeeperConnection Watcher:127.0.0.1:47779 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 834548 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 834549 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 834550 T4819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b913b70 name:ZooKeeperConnection Watcher:127.0.0.1:47779/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 834550 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 834551 T4793 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 834554 T4793 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 834556 T4793 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 834557 T4793 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39617_gyu
   [junit4]   2> 834558 T4793 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39617_gyu
   [junit4]   2> 834559 T4793 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 834560 T4793 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 834564 T4793 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39617_gyu
   [junit4]   2> 834565 T4793 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 834567 T4793 oasc.Overseer.start Overseer (id=91312872090173445-127.0.0.1:39617_gyu-n_0000000000) starting
   [junit4]   2> 834568 T4793 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 834571 T4821 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 834571 T4793 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 834571 T4821 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 834573 T4793 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 834574 T4793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 834575 T4820 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 834576 T4793 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 834577 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 834577 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 834578 T4824 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ec84600 name:ZooKeeperConnection Watcher:127.0.0.1:47779/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 834578 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 834580 T4793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 834582 T4793 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 834807 T4793 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 834807 T4793 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 834809 T4793 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42131
   [junit4]   2> 834809 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 834810 T4793 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 834810 T4793 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556
   [junit4]   2> 834810 T4793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/'
   [junit4]   2> 834820 T4793 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solr.xml
   [junit4]   2> 834828 T4793 oasc.CoreContainer.<init> New CoreContainer 530382479
   [junit4]   2> 834828 T4793 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/]
   [junit4]   2> 834829 T4793 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 834829 T4793 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 834829 T4793 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 834830 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 834830 T4793 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 834830 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 834830 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 834831 T4793 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 834831 T4793 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 834833 T4793 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 834833 T4793 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 834833 T4793 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 834834 T4793 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47779/solr
   [junit4]   2> 834834 T4793 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 834835 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 834836 T4835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a0be1a1 name:ZooKeeperConnection Watcher:127.0.0.1:47779 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 834836 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 834837 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 834838 T4837 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e3c7aca name:ZooKeeperConnection Watcher:127.0.0.1:47779/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 834838 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 834841 T4793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 835843 T4793 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42131_gyu
   [junit4]   2> 835845 T4793 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42131_gyu
   [junit4]   2> 835848 T4819 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 835848 T4837 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 835848 T4824 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 835853 T4793 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 835854 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 836112 T4793 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 836112 T4793 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 836114 T4793 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51319
   [junit4]   2> 836114 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 836115 T4793 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 836115 T4793 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393323855829
   [junit4]   2> 836115 T4793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393323855829/'
   [junit4]   2> 836124 T4793 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393323855829/solr.xml
   [junit4]   2> 836132 T4793 oasc.CoreContainer.<init> New CoreContainer 718963599
   [junit4]   2> 836132 T4793 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393323855829/]
   [junit4]   2> 836133 T4793 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 836133 T4793 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 836133 T4793 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 836133 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 836133 T4793 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 836134 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 836134 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 836134 T4793 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 836134 T4793 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 836136 T4793 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 836136 T4793 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 836137 T4793 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 836137 T4793 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47779/solr
   [junit4]   2> 836137 T4793 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 836138 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 836139 T4849 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a52cd38 name:ZooKeeperConnection Watcher:127.0.0.1:47779 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 836139 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 836141 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 836141 T4851 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bda6b9e name:ZooKeeperConnection Watcher:127.0.0.1:47779/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 836142 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 836146 T4793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 837148 T4793 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51319_gyu
   [junit4]   2> 837151 T4793 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51319_gyu
   [junit4]   2> 837152 T4819 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 837153 T4824 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 837152 T4851 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 837153 T4837 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 837155 T4793 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 837156 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 837402 T4793 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 837402 T4793 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 837404 T4793 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52434
   [junit4]   2> 837404 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 837405 T4793 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 837405 T4793 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130
   [junit4]   2> 837405 T4793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/'
   [junit4]   2> 837414 T4793 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solr.xml
   [junit4]   2> 837422 T4793 oasc.CoreContainer.<init> New CoreContainer 1324328752
   [junit4]   2> 837422 T4793 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/]
   [junit4]   2> 837423 T4793 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 837423 T4793 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 837423 T4793 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 837423 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 837424 T4793 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 837424 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 837424 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 837424 T4793 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 837425 T4793 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 837426 T4793 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 837427 T4793 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 837427 T4793 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 837427 T4793 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47779/solr
   [junit4]   2> 837427 T4793 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 837428 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 837429 T4863 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@127ce9b2 name:ZooKeeperConnection Watcher:127.0.0.1:47779 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 837429 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 837431 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 837431 T4865 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f6cd7f8 name:ZooKeeperConnection Watcher:127.0.0.1:47779/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 837432 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 837445 T4793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 838447 T4793 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52434_gyu
   [junit4]   2> 838448 T4793 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52434_gyu
   [junit4]   2> 838450 T4837 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 838450 T4819 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 838450 T4851 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 838450 T4824 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 838450 T4865 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 838453 T4793 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 838454 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 838722 T4793 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 838723 T4793 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 838725 T4793 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46674
   [junit4]   2> 838725 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 838725 T4793 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 838726 T4793 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428
   [junit4]   2> 838726 T4793 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/'
   [junit4]   2> 838736 T4793 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solr.xml
   [junit4]   2> 838744 T4793 oasc.CoreContainer.<init> New CoreContainer 1158466835
   [junit4]   2> 838744 T4793 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/]
   [junit4]   2> 838745 T4793 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 838745 T4793 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 838745 T4793 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 838746 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 838746 T4793 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 838746 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 838746 T4793 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 838747 T4793 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 838747 T4793 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 838749 T4793 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 838749 T4793 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 838749 T4793 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 838750 T4793 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47779/solr
   [junit4]   2> 838750 T4793 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 838751 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 838752 T4877 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@225a4a30 name:ZooKeeperConnection Watcher:127.0.0.1:47779 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 838752 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 838754 T4793 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 838755 T4879 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e1b6659 name:ZooKeeperConnection Watcher:127.0.0.1:47779/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 838755 T4793 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 838758 T4793 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 839759 T4793 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46674_gyu
   [junit4]   2> 839761 T4793 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46674_gyu
   [junit4]   2> 839762 T4865 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 839763 T4851 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 839763 T4879 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 839763 T4824 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 839763 T4819 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 839763 T4837 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 839767 T4793 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 839768 T4793 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 839777 T4832 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&router.name=compositeId&action=CREATE&numShards=2&wt=javabin&version=2
   [junit4]   2> 839777 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839778 T4821 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 839778 T4821 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 839779 T4821 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 839779 T4821 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 839781 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839782 T4820 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 839782 T4820 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 839783 T4820 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 839784 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839784 T4837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 839784 T4879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 839784 T4819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 839784 T4865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 839784 T4851 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 839784 T4824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 839881 T4821 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 839882 T4821 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:46674_gyu
   [junit4]   2> 839883 T4821 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:42131_gyu
   [junit4]   2> 839884 T4821 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:52434_gyu
   [junit4]   2> 839886 T4874 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 839886 T4833 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 839887 T4821 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:39617_gyu
   [junit4]   2> 839887 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839887 T4874 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 839887 T4833 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 839888 T4858 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 839892 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839892 T4820 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 839892 T4812 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 839892 T4858 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 839892 T4820 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46674/gyu",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46674_gyu",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 839893 T4812 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 839895 T4820 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42131/gyu",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42131_gyu",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 839896 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839897 T4820 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52434/gyu",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52434_gyu",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 839898 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839899 T4820 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39617/gyu",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39617_gyu",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 839900 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 840002 T4819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 840002 T4824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 840002 T4865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 840002 T4851 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 840002 T4837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 840002 T4879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 840888 T4874 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 840888 T4833 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 840888 T4874 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solr.xml
   [junit4]   2> 840889 T4874 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solrj_collection_shard1_replica1
   [junit4]   2> 840889 T4874 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 840888 T4833 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solr.xml
   [junit4]   2> 840889 T4833 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solrj_collection_shard1_replica2
   [junit4]   2> 840889 T4874 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 840890 T4874 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 840889 T4833 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 840891 T4874 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 840891 T4833 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 840891 T4874 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solrj_collection_shard1_replica1/'
   [junit4]   2> 840891 T4833 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 840893 T4858 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 840893 T4858 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solr.xml
   [junit4]   2> 840893 T4858 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solrj_collection_shard2_replica1
   [junit4]   2> 840894 T4833 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 840893 T4812 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 840894 T4833 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solrj_collection_shard1_replica2/'
   [junit4]   2> 840894 T4858 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 840894 T4812 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solr.xml
   [junit4]   2> 840895 T4812 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solrj_collection_shard2_replica2
   [junit4]   2> 840895 T4812 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 840896 T4858 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 840896 T4858 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 840896 T4812 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 840896 T4812 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 840897 T4858 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 840897 T4858 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solrj_collection_shard2_replica1/'
   [junit4]   2> 840900 T4812 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 840901 T4812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solrj_collection_shard2_replica2/'
   [junit4]   2> 840937 T4874 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 840939 T4833 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 840940 T4858 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 840945 T4812 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 840952 T4874 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 840952 T4833 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 840954 T4833 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 840956 T4874 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 840958 T4812 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 840959 T4858 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 840960 T4812 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 840960 T4833 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 840961 T4858 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 840963 T4874 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 840966 T4858 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 840968 T4812 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 841087 T4833 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 841093 T4833 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 841095 T4833 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 841105 T4874 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 841124 T4833 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 841124 T4874 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 841126 T4874 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 841127 T4833 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 841130 T4833 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 841132 T4833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 841132 T4833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 841132 T4833 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 841134 T4833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 841134 T4833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 841134 T4833 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 841135 T4833 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solrj_collection_shard1_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solrj_collection_shard1_replica2/data/
   [junit4]   2> 841135 T4833 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b0237b1
   [junit4]   2> 841136 T4833 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solrj_collection_shard1_replica2/data
   [junit4]   2> 841136 T4833 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 841136 T4833 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 841136 T4874 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 841137 T4833 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393323854556/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 841137 T4833 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4276170103600879]
   [junit4]   2> 841138 T4833 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@119f2071 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1712851b),segFN=segments_1,generation=1}
   [junit4]   2> 841138 T4833 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 841139 T4874 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 841141 T4833 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 841141 T4833 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 841141 T4874 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 841141 T4833 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 841142 T4833 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 841142 T4833 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 841142 T4874 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 841143 T4874 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 841143 T4833 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 841143 T4874 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 841144 T4833 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 841144 T4833 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 841144 T4833 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 841144 T4874 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 841145 T4833 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 841145 T4874 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 841145 T4833 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 841146 T4833 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 841145 T4874 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 841146 T4874 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solrj_collection_shard1_replica1/data/
   [junit4]   2> 841147 T4874 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b0237b1
   [junit4]   2> 841147 T4812 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 841146 T4833 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 841148 T4833 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 841148 T4833 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 841148 T4833 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 841149 T4833 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 841150 T4858 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 841150 T4812 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 841150 T4874 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solrj_collection_shard1_replica1/data
   [junit4]   2> 841150 T4833 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 841151 T4874 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 841151 T4874 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 841152 T4874 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393323858428/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 841152 T4812 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 841153 T4833 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 841153 T4833 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 841152 T4874 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4276170103600879]
   [junit4]   2> 841154 T4858 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 841154 T4874 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1342328b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5b9c6707),segFN=segments_1,generation=1}
   [junit4]   2> 841155 T4874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 841154 T4833 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6237509867633402]
   [junit4]   2> 841155 T4833 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@119f2071 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1712851b),segFN=segments_1,generation=1}
   [junit4]   2> 841156 T4833 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 841156 T4833 oass.SolrIndexSearcher.<init> Opening Searcher@23dce093[solrj_collection_shard1_replica2] main
   [junit4]   2> 841155 T4858 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 841159 T4885 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@23dce093[solrj_collection_shard1_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 841161 T4833 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 841161 T4833 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:http://127.0.0.1:42131/gyu collection:solrj_collection shard:shard1
   [junit4]   2> 841161 T4874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 841162 T4874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 841163 T4874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 841163 T4874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 841163 T4833 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 841163 T4874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 841164 T4812 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 841164 T4874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 841165 T4874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 841165 T4874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 841166 T4874 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 841166 T4874 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 841167 T4874 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 841167 T4874 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 841167 T4874 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 841168 T4874 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 841168 T4874 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 841168 T4812 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 841168 T4874 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 841169 T4874 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 841170 T4874 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 841171 T4874 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 841172 T4874 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 841172 T4874 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6237509867633402]
   [junit4]   2> 841173 T4874 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1342328b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5b9c6707),segFN=segments_1,generation=1}
   [junit4]   2> 841173 T4874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 841173 T4874 oass.SolrIndexSearcher.<init> Opening Searcher@19969cbd[solrj_collection_shard1_replica1] main
   [junit4]   2> 841175 T4887 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@19969cbd[solrj_collection_shard1_replica1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 841176 T4874 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
   [junit4]   2> 841176 T4874 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:http://127.0.0.1:46674/gyu collection:solrj_collection shard:shard1
   [junit4]   2> 841194 T4874 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 841196 T4812 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 841198 T4812 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 841198 T4812 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 841199 T4812 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 841199 T4833 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 841200 T4812 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 841201 T4812 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 841201 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 841200 T4858 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 841201 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 841201 T4812 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 841202 T4833 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 841202 T4833 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 841202 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C1638 name=solrj_collection_shard1_replica2 org.apache.solr.core.SolrCore@528d10ca url=http://127.0.0.1:42131/gyu/solrj_collection_shard1_replica2 node=127.0.0.1:42131_gyu C1638_STATE=coll:solrj_collection core:solrj_collection_shard1_replica2 props:{state=down, base_url=http://127.0.0.1:42131/gyu, core=solrj_collection_shard1_replica2, node_name=127.0.0.1:42131_gyu}
   [junit4]   2> 841203 T4833 C1638 P42131 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42131/gyu/solrj_collection_shard1_replica2/
   [junit4]   2> 841202 T4812 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solrj_collection_shard2_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solrj_collection_shard2_replica2/data/
   [junit4]   2> 841203 T4833 C1638 P42131 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica2 url=http://127.0.0.1:42131/gyu START replicas=[http://127.0.0.1:46674/gyu/solrj_collection_shard1_replica1/] nUpdates=100
   [junit4]   2> 841203 T4820 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 841204 T4833 C1638 P42131 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica2 url=http://127.0.0.1:42131/gyu DONE.  We have no versions.  sync failed.
   [junit4]   2> 841204 T4833 C1638 P42131 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 841205 T4833 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 841205 T4833 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42131/gyu/solrj_collection_shard1_replica2/ shard1
   [junit4]   2> 841205 T4833 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard1
   [junit4]   2> 841203 T4812 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b0237b1
   [junit4]   2> 841205 T4858 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> ASYNC  NEW_CORE C1639 name=solrj_collection_shard1_replica1 org.apache.solr.core.SolrCore@5851b83e url=http://127.0.0.1:46674/gyu/solrj_collection_shard1_replica1 node=127.0.0.1:46674_gyu C1639_STATE=coll:solrj_collection core:solrj_collection_shard1_replica1 props:{state=down, base_url=http://127.0.0.1:46674/gyu, core=solrj_collection_shard1_replica1, node_name=127.0.0.1:46674_gyu}
   [junit4]   2> 841207 T4873 C1639 P46674 oasc.SolrCore.execute [solrj_collection_shard1_replica1] webapp=/gyu path=/get params={distrib=false&qt=/get&getVersions=100&wt=javabin&version=2} status=0 QTime=0 
   [junit4]   2> 841208 T4812 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solrj_collection_shard2_replica2/data
   [junit4]   2> 841208 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 841209 T4812 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 841209 T4812 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 841210 T4812 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393323854248/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 841211 T4812 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4276170103600879]
   [junit4]   2> 841211 T4858 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 841211 T4812 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@73c3fca7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@22d9327e),segFN=segments_1,generation=1}
   [junit4]   2> 841212 T4812 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 841212 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 841212 T4858 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 841213 T4858 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 841213 T4858 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 841214 T4858 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 841214 T4858 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 841215 T4858 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 841215 T4858 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solrj_collection_shard2_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solrj_collection_shard2_replica1/data/
   [junit4]   2> 841215 T4819 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 841215 T4858 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b0237b1
   [junit4]   2> 841216 T4858 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solrj_collection_shard2_replica1/data
   [junit4]   2> 841217 T4858 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 841217 T4858 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 841217 T4858 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393323857130/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 841218 T4858 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4276170103600879]
   [junit4]   2> 841216 T4812 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 841218 T4858 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@45e82a43 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@58832791),segFN=segments_1,generation=1}
   [junit4]   2> 841219 T4858 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 841218 T4812 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 841219 T4812 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 841220 T4812 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 841220 T4812 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 841220 T4812 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 841221 T4812 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 841221 T4858 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 841221 T4812 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 841222 T4858 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 841222 T4812 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 841222 T4858 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 841223 T4812 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 841223 T4858 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 841223 T4812 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 841224 T4812 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 841223 T4858 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 841224 T4812 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 841224 T4858 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 841225 T4812 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 841225

[...truncated too long message...]

er:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:414)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 939792 T4851 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_1/leader_elect/shard1/election/91312872090173450-core_node2-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 939793 T4851 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 939793 T4851 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 939793 T4851 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 939793 T4851 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 939793 T4851 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 939794 T4851 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 939794 T4851 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 939794 T4851 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 939794 T4851 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 939794 T4851 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> NOTE: test params are: codec=Lucene46: {other_tl1=Pulsing41(freqCutoff=19 minBlockSize=53 maxBlockSize=179), _version_=Pulsing41(freqCutoff=19 minBlockSize=53 maxBlockSize=179), multiDefault=PostingsFormat(name=Memory doPackFST= true), a_t=Pulsing41(freqCutoff=19 minBlockSize=53 maxBlockSize=179), a_si=PostingsFormat(name=Memory doPackFST= true), intDefault=Pulsing41(freqCutoff=19 minBlockSize=53 maxBlockSize=179), range_facet_sl=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), range_facet_si=PostingsFormat(name=Memory doPackFST= true), id=PostingsFormat(name=MockRandom), text=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), range_facet_l=Pulsing41(freqCutoff=19 minBlockSize=53 maxBlockSize=179), timestamp=Pulsing41(freqCutoff=19 minBlockSize=53 maxBlockSize=179)}, docValues:{timestamp=DocValuesFormat(name=Memory)}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=tr_TR, timezone=Europe/Andorra
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/Oracle Corporation 1.8.0 (64-bit)/cpus=8,threads=1,free=259507768,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [SOLR749Test, JsonLoaderTest, LeaderElectionTest, TermVectorComponentTest, OverseerRolesTest, SchemaVersionSpecificBehaviorTest, OverseerCollectionProcessorTest, SynonymTokenizerTest, TestReplicationHandler, TestJmxMonitoredMap, DistributedTermsComponentTest, TestRandomDVFaceting, LegacyHTMLStripCharFilterTest, HdfsSyncSliceTest, TestReloadAndDeleteDocs, AbstractAnalyticsStatsTest, CursorMarkTest, UUIDFieldTest, TestSort, BJQParserTest, SpatialFilterTest, HdfsCollectionsAPIDistributedZkTest, TestFunctionQuery, TestClassNameShortening, TestConfig, DocValuesTest, ShowFileRequestHandlerTest, ShardRoutingTest, TestSolrQueryParser, SimpleFacetsTest, DirectUpdateHandlerOptimizeTest, TestRemoteStreaming, TestSchemaNameResource, SliceStateUpdateTest, TestShardHandlerFactory, TestMaxScoreQueryParser, SliceStateTest, ReturnFieldsTest, TestCollationFieldDocValues, TestExtendedDismaxParser, ResourceLoaderTest, StatelessScriptUpdateProcessorFactoryTest, ZkCLITest, SampleTest, TestQueryUtils, ShardSplitTest, TestPostingsSolrHighlighter, SolrIndexSplitterTest, TestRecoveryHdfs, UpdateRequestProcessorFactoryTest, CoreAdminHandlerTest, SolrCoreTest, OverseerTest, TestDefaultSearchFieldResource, TestSolrXMLSerializer, TermsComponentTest, WordBreakSolrSpellCheckerTest, TestCloudManagedSchemaAddField, MBeansHandlerTest, IndexSchemaTest, TestImplicitCoreProperties, TestModifyConfFiles, DocumentBuilderTest, DOMUtilTest, BasicFunctionalityTest, TestFreeTextSuggestions, TestManagedSchema, TestFieldTypeCollectionResource, ClusterStateUpdateTest, TestBadConfig, TestWriterPerf, TestSchemaSimilarityResource, TestDocSet, TestGroupingSearch, TestPseudoReturnFields, LoggingHandlerTest, PeerSyncTest, SuggesterFSTTest, FieldFacetExtrasTest, HdfsChaosMonkeySafeLeaderTest, HdfsRecoveryZkTest, URLClassifyProcessorTest, TestCoreContainer, DeleteShardTest, TestNRTOpen, TestRealTimeGet, PrimitiveFieldTypeTest, TestBM25SimilarityFactory, HdfsLockFactoryTest, AssignTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J0 in 106.17s, 1 test, 1 failure <<< FAILURES!

[...truncated 703 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1606 tests, 1 failure, 35 ignored (7 assumptions)

Total time: 47 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-fcs-b129 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_51) - Build # 9589 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9589/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Could not find collection:addReplicaColl

Stack Trace:
java.lang.AssertionError: Could not find collection:addReplicaColl
	at __randomizedtesting.SeedInfo.seed([53A0571F37BBA1D6:D246D90740E4C1EA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:147)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:724)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1193)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10422 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 638263 T2134 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 638263 T2134 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /xnts/
   [junit4]   2> 638267 T2134 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1393317744345
   [junit4]   2> 638267 T2134 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 638268 T2135 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 638368 T2134 oasc.ZkTestServer.run start zk server on port:47963
   [junit4]   2> 638369 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 638376 T2141 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44ac8bca name:ZooKeeperConnection Watcher:127.0.0.1:47963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 638376 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 638376 T2134 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 638379 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 638381 T2143 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b9161ac name:ZooKeeperConnection Watcher:127.0.0.1:47963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 638381 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 638381 T2134 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 638383 T2134 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 638385 T2134 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 638387 T2134 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 638389 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 638389 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 638393 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 638393 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 638395 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 638396 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 638398 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 638398 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 638401 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 638401 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 638403 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 638403 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 638406 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 638406 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 638410 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 638410 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 638412 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 638412 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 638415 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 638415 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 638417 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 638417 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 638420 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 638421 T2145 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11cd7671 name:ZooKeeperConnection Watcher:127.0.0.1:47963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 638422 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 638422 T2134 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 638424 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 638425 T2147 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@583063ab name:ZooKeeperConnection Watcher:127.0.0.1:47963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 638425 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 638426 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 638426 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 638429 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 638429 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 638432 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 638432 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 638434 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 638434 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 638437 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 638438 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 638443 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 638444 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 638449 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 638449 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 638453 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 638453 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 638457 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 638457 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 638463 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 638464 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 638470 T2134 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 638472 T2134 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 638743 T2134 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 638747 T2134 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42170
   [junit4]   2> 638747 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 638747 T2134 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 638748 T2134 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559
   [junit4]   2> 638748 T2134 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/'
   [junit4]   2> 638757 T2134 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solr.xml
   [junit4]   2> 638792 T2134 oasc.CoreContainer.<init> New CoreContainer 112044092
   [junit4]   2> 638792 T2134 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/]
   [junit4]   2> 638793 T2134 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 638794 T2134 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 638794 T2134 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 638794 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 638794 T2134 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 638795 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 638795 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 638795 T2134 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 638795 T2134 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 638797 T2134 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 638798 T2134 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 638798 T2134 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 638798 T2134 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47963/solr
   [junit4]   2> 638798 T2134 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 638799 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 638810 T2158 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72fffe90 name:ZooKeeperConnection Watcher:127.0.0.1:47963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 638811 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 638815 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 638817 T2160 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@698a8b2c name:ZooKeeperConnection Watcher:127.0.0.1:47963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 638818 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 638820 T2134 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 638823 T2134 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 638827 T2134 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 638829 T2134 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42170_xnts
   [junit4]   2> 638830 T2134 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42170_xnts
   [junit4]   2> 638833 T2134 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 638834 T2134 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 638838 T2134 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42170_xnts
   [junit4]   2> 638838 T2134 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 638840 T2134 oasc.Overseer.start Overseer (id=91312471693656069-127.0.0.1:42170_xnts-n_0000000000) starting
   [junit4]   2> 638842 T2134 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 638846 T2162 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 638847 T2134 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 638847 T2162 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 638849 T2134 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 638851 T2134 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 638853 T2161 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 638855 T2134 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 638855 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 638856 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 638857 T2165 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fa96854 name:ZooKeeperConnection Watcher:127.0.0.1:47963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 638857 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 638858 T2134 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 638859 T2134 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 639067 T2134 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 639068 T2134 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 639070 T2134 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47860
   [junit4]   2> 639071 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 639071 T2134 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 639071 T2134 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938
   [junit4]   2> 639071 T2134 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/'
   [junit4]   2> 639090 T2134 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solr.xml
   [junit4]   2> 639136 T2134 oasc.CoreContainer.<init> New CoreContainer 1588797618
   [junit4]   2> 639137 T2134 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/]
   [junit4]   2> 639138 T2134 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 639138 T2134 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 639138 T2134 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 639139 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 639139 T2134 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 639139 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 639139 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 639139 T2134 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 639140 T2134 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 639142 T2134 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 639143 T2134 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 639143 T2134 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 639143 T2134 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47963/solr
   [junit4]   2> 639143 T2134 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 639144 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 639146 T2176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4227b6b0 name:ZooKeeperConnection Watcher:127.0.0.1:47963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 639147 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 639150 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 639151 T2178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b6263e3 name:ZooKeeperConnection Watcher:127.0.0.1:47963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 639152 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 639157 T2134 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 640161 T2134 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47860_xnts
   [junit4]   2> 640162 T2134 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47860_xnts
   [junit4]   2> 640164 T2165 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 640165 T2178 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 640165 T2160 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 640170 T2134 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 640170 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 640379 T2134 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 640380 T2134 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 640382 T2134 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56229
   [junit4]   2> 640382 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 640382 T2134 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 640383 T2134 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249
   [junit4]   2> 640383 T2134 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/'
   [junit4]   2> 640392 T2134 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solr.xml
   [junit4]   2> 640438 T2134 oasc.CoreContainer.<init> New CoreContainer 59923308
   [junit4]   2> 640438 T2134 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/]
   [junit4]   2> 640439 T2134 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 640440 T2134 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 640440 T2134 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 640440 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 640440 T2134 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 640440 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 640441 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 640441 T2134 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 640441 T2134 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 640443 T2134 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 640443 T2134 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 640443 T2134 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 640444 T2134 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47963/solr
   [junit4]   2> 640444 T2134 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 640445 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 640446 T2190 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7db01148 name:ZooKeeperConnection Watcher:127.0.0.1:47963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 640447 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 640461 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 640462 T2192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6007dfdb name:ZooKeeperConnection Watcher:127.0.0.1:47963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 640462 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 640467 T2134 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 641470 T2134 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56229_xnts
   [junit4]   2> 641472 T2134 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56229_xnts
   [junit4]   2> 641478 T2165 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 641478 T2178 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 641478 T2192 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 641478 T2160 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 641488 T2134 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 641489 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 641717 T2134 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 641718 T2134 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 641720 T2134 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43791
   [junit4]   2> 641721 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 641721 T2134 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 641722 T2134 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568
   [junit4]   2> 641722 T2134 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/'
   [junit4]   2> 641732 T2134 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/solr.xml
   [junit4]   2> 641775 T2134 oasc.CoreContainer.<init> New CoreContainer 1900985541
   [junit4]   2> 641775 T2134 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/]
   [junit4]   2> 641776 T2134 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 641777 T2134 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 641777 T2134 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 641777 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 641777 T2134 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 641777 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 641778 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 641778 T2134 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 641778 T2134 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 641780 T2134 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 641780 T2134 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 641780 T2134 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 641781 T2134 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47963/solr
   [junit4]   2> 641781 T2134 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 641782 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 641783 T2204 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@684b63a0 name:ZooKeeperConnection Watcher:127.0.0.1:47963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 641783 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 641785 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 641786 T2206 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@463abeb4 name:ZooKeeperConnection Watcher:127.0.0.1:47963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 641786 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 641792 T2134 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 642795 T2134 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43791_xnts
   [junit4]   2> 642796 T2134 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43791_xnts
   [junit4]   2> 642801 T2160 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 642802 T2206 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 642801 T2178 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 642802 T2192 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 642803 T2165 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 642816 T2134 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 642817 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 643053 T2134 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 643053 T2134 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 643056 T2134 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52701
   [junit4]   2> 643057 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 643057 T2134 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 643058 T2134 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393317748895
   [junit4]   2> 643058 T2134 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393317748895/'
   [junit4]   2> 643073 T2134 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393317748895/solr.xml
   [junit4]   2> 643109 T2134 oasc.CoreContainer.<init> New CoreContainer 326589746
   [junit4]   2> 643109 T2134 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393317748895/]
   [junit4]   2> 643110 T2134 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 643110 T2134 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 643111 T2134 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 643111 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 643111 T2134 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 643111 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 643111 T2134 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 643112 T2134 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 643112 T2134 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 643114 T2134 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 643114 T2134 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 643114 T2134 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 643114 T2134 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47963/solr
   [junit4]   2> 643115 T2134 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 643115 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 643117 T2218 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@187879a name:ZooKeeperConnection Watcher:127.0.0.1:47963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 643117 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 643119 T2134 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 643120 T2220 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5208c07e name:ZooKeeperConnection Watcher:127.0.0.1:47963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 643120 T2134 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 643123 T2134 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 644127 T2134 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52701_xnts
   [junit4]   2> 644131 T2134 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52701_xnts
   [junit4]   2> 644136 T2206 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 644136 T2160 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 644136 T2178 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 644136 T2192 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 644137 T2165 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 644137 T2220 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 644147 T2134 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 644148 T2134 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 644165 T2172 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&name=solrj_collection&wt=javabin&router.field=myOwnField&collection.configName=conf1&numShards=2&version=2&router.name=compositeId&action=CREATE
   [junit4]   2> 644167 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644169 T2162 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 644170 T2162 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 644171 T2162 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 644171 T2162 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 644178 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644180 T2161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 644181 T2161 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 644181 T2161 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 644185 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644186 T2206 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644186 T2165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644186 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644186 T2192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644187 T2220 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644187 T2178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644278 T2162 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 644278 T2162 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:47860_xnts
   [junit4]   2> 644280 T2162 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:56229_xnts
   [junit4]   2> 644282 T2162 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:43791_xnts
   [junit4]   2> 644284 T2171 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 644285 T2162 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:42170_xnts
   [junit4]   2> 644286 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644287 T2171 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644287 T2185 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 644287 T2202 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 644290 T2185 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644292 T2202 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644292 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644292 T2154 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 644293 T2161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 644293 T2154 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644294 T2161 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47860/xnts",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47860_xnts",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644296 T2161 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56229/xnts",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56229_xnts",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644306 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644307 T2161 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43791/xnts",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43791_xnts",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644309 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644311 T2161 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42170/xnts",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42170_xnts",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644313 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644416 T2220 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644416 T2192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644416 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644416 T2165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644416 T2178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644417 T2206 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 645287 T2171 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 645288 T2171 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solr.xml
   [junit4]   2> 645289 T2171 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solrj_collection_shard1_replica1
   [junit4]   2> 645289 T2171 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 645290 T2185 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 645291 T2171 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645291 T2185 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solr.xml
   [junit4]   2> 645291 T2171 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 645292 T2185 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solrj_collection_shard1_replica2
   [junit4]   2> 645292 T2185 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 645292 T2202 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 645293 T2185 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645294 T2154 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 645294 T2202 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/solr.xml
   [junit4]   2> 645294 T2185 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 645293 T2171 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 645295 T2202 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/solrj_collection_shard2_replica1
   [junit4]   2> 645296 T2202 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 645295 T2154 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solr.xml
   [junit4]   2> 645297 T2154 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solrj_collection_shard2_replica2
   [junit4]   2> 645297 T2154 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 645297 T2185 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 645296 T2171 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solrj_collection_shard1_replica1/'
   [junit4]   2> 645299 T2154 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645298 T2185 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solrj_collection_shard1_replica2/'
   [junit4]   2> 645297 T2202 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645300 T2202 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 645299 T2154 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 645302 T2202 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 645302 T2154 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 645302 T2202 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/solrj_collection_shard2_replica1/'
   [junit4]   2> 645303 T2154 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solrj_collection_shard2_replica2/'
   [junit4]   2> 645369 T2171 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 645370 T2185 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 645371 T2154 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 645389 T2202 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 645438 T2171 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 645439 T2154 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 645440 T2171 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 645441 T2154 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 645455 T2154 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 645455 T2171 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 645459 T2185 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 645470 T2185 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 645474 T2202 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 645477 T2202 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 645481 T2185 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 645484 T2202 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 646025 T2171 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 646030 T2171 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 646042 T2171 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 646066 T2154 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 646068 T2171 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 646079 T2171 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 646080 T2154 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 646084 T2154 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 646084 T2171 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 646085 T2171 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 646086 T2171 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 646086 T2171 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 646087 T2171 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 646087 T2171 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 646088 T2171 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 646088 T2171 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solrj_collection_shard1_replica1/data/
   [junit4]   2> 646088 T2171 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@629956f3
   [junit4]   2> 646089 T2171 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solrj_collection_shard1_replica1/data
   [junit4]   2> 646090 T2171 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 646090 T2171 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 646091 T2171 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393317744938/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 646091 T2171 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=86.8154296875, floorSegmentMB=0.337890625, forceMergeDeletesPctAllowed=8.945956538783088, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.14696219176497183
   [junit4]   2> 646091 T2185 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 646092 T2171 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@37e14265 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36f944eb),segFN=segments_1,generation=1}
   [junit4]   2> 646092 T2171 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 646096 T2171 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 646096 T2154 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 646097 T2185 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 646097 T2171 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 646097 T2171 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 646098 T2171 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 646098 T2171 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 646098 T2171 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 646099 T2171 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 646099 T2171 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 646100 T2185 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 646100 T2171 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 646100 T2154 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 646101 T2171 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 646101 T2171 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 646102 T2171 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 646102 T2171 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 646102 T2171 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 646103 T2171 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 646103 T2171 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 646104 T2171 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 646104 T2154 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 646105 T2154 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 646106 T2154 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 646106 T2154 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 646106 T2171 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 646107 T2154 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 646107 T2154 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 646113 T2154 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 646114 T2154 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solrj_collection_shard2_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solrj_collection_shard2_replica2/data/
   [junit4]   2> 646114 T2154 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@629956f3
   [junit4]   2> 646115 T2202 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 646115 T2171 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 646115 T2171 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 646115 T2154 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solrj_collection_shard2_replica2/data
   [junit4]   2> 646115 T2171 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=1.658203125, floorSegmentMB=2.0087890625, forceMergeDeletesPctAllowed=22.454872126260824, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6954425857206201
   [junit4]   2> 646115 T2154 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 646116 T2171 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@37e14265 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36f944eb),segFN=segments_1,generation=1}
   [junit4]   2> 646116 T2171 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 646116 T2154 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 646117 T2171 oass.SolrIndexSearcher.<init> Opening Searcher@2c2eec66[solrj_collection_shard1_replica1] main
   [junit4]   2> 646117 T2154 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393317744559/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 646117 T2154 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=86.8154296875, floorSegmentMB=0.337890625, forceMergeDeletesPctAllowed=8.945956538783088, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.14696219176497183
   [junit4]   2> 646118 T2185 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 646118 T2154 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3ca23333 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f577a0b),segFN=segments_1,generation=1}
   [junit4]   2> 646119 T2154 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 646119 T2226 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@2c2eec66[solrj_collection_shard1_replica1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 646119 T2202 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 646120 T2171 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
   [junit4]   2> 646120 T2171 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:http://127.0.0.1:47860/xnts collection:solrj_collection shard:shard1
   [junit4]   2> 646122 T2202 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 646122 T2171 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 646122 T2185 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 646122 T2154 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 646123 T2154 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 646123 T2154 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 646123 T2154 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 646124 T2154 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 646124 T2154 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 646125 T2154 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 646125 T2154 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 646125 T2154 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 646126 T2154 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 646126 T2185 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 646127 T2154 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 646127 T2154 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 646127 T2171 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 646127 T2154 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 646128 T2185 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 646128 T2185 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 646128 T2154 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 646128 T2185 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 646129 T2154 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 646129 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646129 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646129 T2154 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 646129 T2171 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=0
   [junit4]   2> 646130 T2154 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 646129 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646130 T2171 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
   [junit4]   2> 646130 T2171 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 646130 T2185 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 646131 T2185 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> ASYNC  NEW_CORE C3278 name=solrj_collection_shard1_replica1 org.apache.solr.core.SolrCore@62f6a606 url=http://127.0.0.1:47860/xnts/solrj_collection_shard1_replica1 node=127.0.0.1:47860_xnts C3278_STATE=coll:solrj_collection core:solrj_collection_shard1_replica1 props:{state=down, base_url=http://127.0.0.1:47860/xnts, core=solrj_collection_shard1_replica1, node_name=127.0.0.1:47860_xnts}
   [junit4]   2> 646131 T2171 C3278 P47860 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47860/xnts/solrj_collection_shard1_replica1/
   [junit4]   2> 646130 T2161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 646131 T2171 C3278 P47860 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=http://127.0.0.1:47860/xnts START replicas=[http://127.0.0.1:56229/xnts/solrj_collection_shard1_replica2/] nUpdates=100
   [junit4]   2> 646131 T2185 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 646131 T2154 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 646132 T2185 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solrj_collection_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solrj_collection_shard1_replica2/data/
   [junit4]   2> 646132 T2171 C3278 P47860 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=http://127.0.0.1:47860/xnts DONE.  We have no versions.  sync failed.
   [junit4]   2> 646134 T2171 C3278 P47860 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 646133 T2202 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 646133 T2185 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@629956f3
   [junit4]   2> 646134 T2171 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 646135 T2171 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47860/xnts/solrj_collection_shard1_replica1/ shard1
   [junit4]   2> 646134 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646135 T2185 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solrj_collection_shard1_replica2/data
   [junit4]   2> 646135 T2185 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 646135 T2171 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard1
   [junit4]   2> 646136 T2154 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 646136 T2185 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 646136 T2178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646136 T2220 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646136 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646135 T2165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646135 T2192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646135 T2206 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646138 T2202 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 646137 T2185 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393317746249/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 646137 T2154 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 646139 T2185 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=86.8154296875, floorSegmentMB=0.337890625, forceMergeDeletesPctAllowed=8.945956538783088, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.14696219176497183
   [junit4]   2> 646139 T2154 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=1.658203125, floorSegmentMB=2.0087890625, forceMergeDeletesPctAllowed=22.454872126260824, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6954425857206201
   [junit4]   2> 646140 T2185 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2489da21 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@38ee7aa8),segFN=segments_1,generation=1}
   [junit4]   2> 646140 T2185 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 646140 T2154 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3ca23333 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f577a0b),segFN=segments_1,generation=1}
   [junit4]   2> 646141 T2154 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 646141 T2154 oass.SolrIndexSearcher.<init> Opening Searcher@29365459[solrj_collection_shard2_replica2] main
   [junit4]   2> 646141 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646142 T2202 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 646143 T2161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 646144 T2229 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@29365459[solrj_collection_shard2_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 646144 T2185 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 646144 T2202 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 646145 T2202 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 646145 T2185 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 646145 T2154 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
   [junit4]   2> 646145 T2202 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 646145 T2154 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica2 address:http://127.0.0.1:42170/xnts collection:solrj_collection shard:shard2
   [junit4]   2> 646145 T2185 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 646146 T2185 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 646146 T2160 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646146 T2185 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 646147 T2202 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 646147 T2154 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 646147 T2202 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 646147 T2185 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 646148 T2202 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 646148 T2185 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 646148 T2202 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/solrj_collection_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/solrj_collection_shard2_replica1/data/
   [junit4]   2> 646148 T2185 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 646148 T2202 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@629956f3
   [junit4]   2> 646149 T2185 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 646149 T2185 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 646149 T2202 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/solrj_collection_shard2_replica1/data
   [junit4]   2> 646150 T2185 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 646150 T2185 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 646150 T2202 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393317747568/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 646150 T2185 oasc.RequestHandlers.

[...truncated too long message...]

keeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:414)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 724778 T2220 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_1/leader_elect/shard3/election/91312471693656078-core_node6-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 724779 T2220 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 724779 T2220 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 724779 T2220 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 724779 T2220 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/nodes_used_collection/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 724780 T2220 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=ko, timezone=Africa/Dar_es_Salaam
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=17968112,total=170676224
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsBasicDistributedZkTest, HardAutoCommitTest, CurrencyFieldOpenExchangeTest, SolrRequestParserTest, ZkNodePropsTest, CoreMergeIndexesAdminHandlerTest, SolrPluginUtilsTest, TestFuzzyAnalyzedSuggestions, FileUtilsTest, HdfsBasicDistributedZk2Test, TestManagedSchemaFieldResource, ConvertedLegacyTest, TestQuerySenderNoQuery, DirectUpdateHandlerTest, TestCSVLoader, BasicZkTest, ShardRoutingCustomTest, TestDynamicFieldCollectionResource, SuggesterWFSTTest, CurrencyFieldXmlFileTest, DirectSolrSpellCheckerTest, SuggesterTSTTest, OutputWriterTest, TestFiltering, BasicDistributedZk2Test, TestFieldResource, OpenExchangeRatesOrgProviderTest, TestSchemaSimilarityResource, TestLazyCores, TestXIncludeConfig, PreAnalyzedUpdateProcessorTest, TestComponentsName, MultiTermTest, NotRequiredUniqueKeyTest, TestIndexingPerformance, AssignTest, TestJmxIntegration, DistanceFunctionTest, TestDFRSimilarityFactory, FullSolrCloudDistribCmdsTest, TestBinaryField, TestSolrXmlPersistence, TestFieldTypeResource, ConnectionManagerTest, SolrTestCaseJ4Test, ShardSplitTest, TestNumberUtils, StandardRequestHandlerTest, CursorMarkTest, SolrCoreCheckLockOnStartupTest, TestDefaultSimilarityFactory, NoCacheHeaderTest, ZkSolrClientTest, TestStressReorder, TestStressLucene, DeleteReplicaTest, RAMDirectoryFactoryTest, OverseerRolesTest, TestCollationFieldDocValues, TestIndexSearcher, TestSolr4Spatial, OpenCloseCoreStressTest, TestStressRecovery, DebugComponentTest, PreAnalyzedFieldTest, IndexSchemaRuntimeFieldTest, DocumentAnalysisRequestHandlerTest, PathHierarchyTokenizerFactoryTest, UniqFieldsUpdateProcessorFactoryTest, DisMaxRequestHandlerTest, TestSweetSpotSimilarityFactory, TestSimpleQParserPlugin, TestSolrXml, TestSolrQueryParserDefaultOperatorResource, RequestHandlersTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 87.54s, 1 test, 1 failure <<< FAILURES!

[...truncated 616 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1606 tests, 1 failure, 35 ignored (7 assumptions)

Total time: 46 minutes 38 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_51) - Build # 9588 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9588/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Could not find collection:addReplicaColl

Stack Trace:
java.lang.AssertionError: Could not find collection:addReplicaColl
	at __randomizedtesting.SeedInfo.seed([C3DB0FFB18D046C7:423D81E36F8F26FB]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:147)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:724)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1193)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10218 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 393166 T1576 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 393166 T1576 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /z_/rj
   [junit4]   2> 393170 T1576 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1393305163737
   [junit4]   2> 393171 T1576 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 393171 T1577 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 393271 T1576 oasc.ZkTestServer.run start zk server on port:53555
   [junit4]   2> 393272 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 393322 T1583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@692f5a6f name:ZooKeeperConnection Watcher:127.0.0.1:53555 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 393323 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 393323 T1576 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 393330 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 393333 T1585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28518cf0 name:ZooKeeperConnection Watcher:127.0.0.1:53555/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 393334 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 393334 T1576 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 393339 T1576 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 393343 T1576 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 393347 T1576 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 393351 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 393352 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 393358 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 393359 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 393364 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 393364 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 393369 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 393369 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 393374 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 393375 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 393378 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 393379 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 393383 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 393384 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 393389 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 393390 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 393396 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 393396 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 393401 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 393401 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 393405 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 393406 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 393412 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 393414 T1587 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@394d3a5d name:ZooKeeperConnection Watcher:127.0.0.1:53555 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 393414 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 393415 T1576 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 393420 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 393422 T1589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bc49920 name:ZooKeeperConnection Watcher:127.0.0.1:53555/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 393422 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 393423 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 393424 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 393431 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 393432 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 393437 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 393437 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 393442 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 393442 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 393447 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 393447 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 393452 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 393452 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 393457 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 393457 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 393463 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 393464 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 393469 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 393469 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 393474 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 393475 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 393480 T1576 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 393480 T1576 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 393706 T1576 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 393710 T1576 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53141
   [junit4]   2> 393711 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 393711 T1576 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 393712 T1576 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054
   [junit4]   2> 393712 T1576 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/'
   [junit4]   2> 393729 T1576 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solr.xml
   [junit4]   2> 393761 T1576 oasc.CoreContainer.<init> New CoreContainer 317517101
   [junit4]   2> 393761 T1576 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/]
   [junit4]   2> 393762 T1576 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 393762 T1576 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 393762 T1576 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 393762 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 393763 T1576 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 393763 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 393763 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 393763 T1576 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 393763 T1576 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 393766 T1576 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 393766 T1576 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 393766 T1576 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 393766 T1576 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53555/solr
   [junit4]   2> 393767 T1576 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 393767 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 393769 T1600 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c72f10 name:ZooKeeperConnection Watcher:127.0.0.1:53555 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 393770 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 393773 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 393775 T1602 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@527c8f3b name:ZooKeeperConnection Watcher:127.0.0.1:53555/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 393775 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 393777 T1576 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 393782 T1576 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 393787 T1576 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 393789 T1576 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53141_z_%2Frj
   [junit4]   2> 393790 T1576 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53141_z_%2Frj
   [junit4]   2> 393795 T1576 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 393798 T1576 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 393805 T1576 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53141_z_%2Frj
   [junit4]   2> 393806 T1576 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 393810 T1576 oasc.Overseer.start Overseer (id=91311647210930181-127.0.0.1:53141_z_%2Frj-n_0000000000) starting
   [junit4]   2> 393815 T1576 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 393821 T1604 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 393822 T1576 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 393823 T1604 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 393826 T1576 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 393829 T1576 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 393833 T1603 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 393847 T1576 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 393848 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 393849 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 393853 T1607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67106765 name:ZooKeeperConnection Watcher:127.0.0.1:53555/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 393854 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 393856 T1576 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 393860 T1576 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 394071 T1576 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 394071 T1576 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 394073 T1576 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53901
   [junit4]   2> 394074 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 394074 T1576 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 394074 T1576 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427
   [junit4]   2> 394074 T1576 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/'
   [junit4]   2> 394083 T1576 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/solr.xml
   [junit4]   2> 394116 T1576 oasc.CoreContainer.<init> New CoreContainer 1954295537
   [junit4]   2> 394116 T1576 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/]
   [junit4]   2> 394117 T1576 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 394118 T1576 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 394118 T1576 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 394118 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 394118 T1576 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 394118 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 394119 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 394119 T1576 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 394119 T1576 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 394121 T1576 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 394122 T1576 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 394122 T1576 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 394122 T1576 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53555/solr
   [junit4]   2> 394123 T1576 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 394123 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 394126 T1618 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ee0bf6 name:ZooKeeperConnection Watcher:127.0.0.1:53555 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 394126 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 394128 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 394129 T1620 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a9928 name:ZooKeeperConnection Watcher:127.0.0.1:53555/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 394129 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 394133 T1576 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 395136 T1576 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53901_z_%2Frj
   [junit4]   2> 395138 T1576 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53901_z_%2Frj
   [junit4]   2> 395143 T1620 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 395143 T1607 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 395143 T1602 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 395152 T1576 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 395153 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 395352 T1576 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 395353 T1576 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 395356 T1576 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37321
   [junit4]   2> 395358 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 395358 T1576 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 395359 T1576 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721
   [junit4]   2> 395359 T1576 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/'
   [junit4]   2> 395371 T1576 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solr.xml
   [junit4]   2> 395403 T1576 oasc.CoreContainer.<init> New CoreContainer 285297565
   [junit4]   2> 395403 T1576 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/]
   [junit4]   2> 395404 T1576 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 395404 T1576 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 395404 T1576 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 395404 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 395405 T1576 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 395405 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 395405 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 395405 T1576 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 395405 T1576 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 395408 T1576 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 395408 T1576 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 395408 T1576 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 395408 T1576 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53555/solr
   [junit4]   2> 395409 T1576 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 395409 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 395411 T1632 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c0b3923 name:ZooKeeperConnection Watcher:127.0.0.1:53555 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 395412 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 395414 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 395415 T1634 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70ad9ca8 name:ZooKeeperConnection Watcher:127.0.0.1:53555/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 395415 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 395420 T1576 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 396426 T1576 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37321_z_%2Frj
   [junit4]   2> 396427 T1576 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37321_z_%2Frj
   [junit4]   2> 396433 T1607 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 396433 T1634 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 396433 T1620 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 396433 T1602 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 396442 T1576 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 396443 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 396651 T1576 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 396652 T1576 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 396656 T1576 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42878
   [junit4]   2> 396657 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 396657 T1576 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 396657 T1576 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011
   [junit4]   2> 396657 T1576 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/'
   [junit4]   2> 396666 T1576 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solr.xml
   [junit4]   2> 396697 T1576 oasc.CoreContainer.<init> New CoreContainer 1822808792
   [junit4]   2> 396698 T1576 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/]
   [junit4]   2> 396699 T1576 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 396699 T1576 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 396699 T1576 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 396699 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 396699 T1576 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 396700 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 396700 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 396700 T1576 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 396700 T1576 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 396702 T1576 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 396703 T1576 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 396703 T1576 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 396703 T1576 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53555/solr
   [junit4]   2> 396703 T1576 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 396704 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 396706 T1646 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cc2d2a1 name:ZooKeeperConnection Watcher:127.0.0.1:53555 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 396706 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 396723 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 396726 T1648 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ca4dc0e name:ZooKeeperConnection Watcher:127.0.0.1:53555/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 396726 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 396817 T1576 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 397822 T1576 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42878_z_%2Frj
   [junit4]   2> 397823 T1576 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42878_z_%2Frj
   [junit4]   2> 397826 T1648 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 397826 T1620 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 397826 T1602 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 397826 T1634 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 397826 T1607 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 397831 T1576 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 397832 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 398043 T1576 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 398044 T1576 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 398046 T1576 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45966
   [junit4]   2> 398047 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 398047 T1576 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 398047 T1576 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399
   [junit4]   2> 398047 T1576 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399/'
   [junit4]   2> 398057 T1576 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399/solr.xml
   [junit4]   2> 398088 T1576 oasc.CoreContainer.<init> New CoreContainer 1521204875
   [junit4]   2> 398088 T1576 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399/]
   [junit4]   2> 398089 T1576 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 398090 T1576 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 398090 T1576 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 398090 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 398090 T1576 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 398090 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 398091 T1576 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 398091 T1576 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 398091 T1576 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 398093 T1576 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 398093 T1576 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 398094 T1576 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 398094 T1576 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53555/solr
   [junit4]   2> 398094 T1576 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 398095 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 398096 T1660 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@416ac62e name:ZooKeeperConnection Watcher:127.0.0.1:53555 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 398097 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 398099 T1576 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 398100 T1662 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cd36b15 name:ZooKeeperConnection Watcher:127.0.0.1:53555/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 398100 T1576 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 398103 T1576 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 399106 T1576 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45966_z_%2Frj
   [junit4]   2> 399108 T1576 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45966_z_%2Frj
   [junit4]   2> 399113 T1662 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 399113 T1634 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 399114 T1648 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 399113 T1607 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 399113 T1620 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 399113 T1602 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 399123 T1576 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 399123 T1576 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 399132 T1614 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&name=solrj_collection&action=CREATE&numShards=2&router.field=myOwnField&router.name=compositeId&version=2&wt=javabin&collection.configName=conf1
   [junit4]   2> 399133 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 399135 T1604 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 399135 T1604 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 399136 T1604 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 399137 T1604 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 399143 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 399145 T1603 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 399145 T1603 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 399146 T1603 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 399149 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 399151 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399151 T1634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399151 T1607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399151 T1662 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399151 T1648 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399151 T1620 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399243 T1604 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 399243 T1604 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:53901_z_%2Frj
   [junit4]   2> 399246 T1604 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:42878_z_%2Frj
   [junit4]   2> 399249 T1604 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:37321_z_%2Frj
   [junit4]   2> 399250 T1616 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 399251 T1604 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:53141_z_%2Frj
   [junit4]   2> 399252 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 399253 T1616 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 399253 T1641 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 399254 T1630 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 399255 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 399255 T1641 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 399257 T1630 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 399257 T1603 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 399258 T1603 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53901/z_/rj",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53901_z_%2Frj",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 399258 T1598 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 399260 T1598 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 399263 T1603 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42878/z_/rj",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42878_z_%2Frj",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 399267 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 399269 T1603 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37321/z_/rj",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37321_z_%2Frj",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 399273 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 399275 T1603 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53141/z_/rj",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53141_z_%2Frj",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 399279 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 399383 T1607 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399383 T1634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399383 T1602 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399383 T1620 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399383 T1648 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 399383 T1662 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 400254 T1616 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 400255 T1616 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/solr.xml
   [junit4]   2> 400255 T1616 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/solrj_collection_shard1_replica1
   [junit4]   2> 400256 T1616 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 400256 T1641 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 400257 T1641 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solr.xml
   [junit4]   2> 400257 T1630 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 400257 T1616 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 400257 T1641 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solrj_collection_shard1_replica2
   [junit4]   2> 400259 T1641 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 400258 T1630 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solr.xml
   [junit4]   2> 400258 T1616 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 400260 T1641 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 400261 T1641 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 400260 T1630 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solrj_collection_shard2_replica1
   [junit4]   2> 400261 T1630 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 400261 T1598 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 400262 T1616 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 400263 T1598 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solr.xml
   [junit4]   2> 400263 T1616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/solrj_collection_shard1_replica1/'
   [junit4]   2> 400263 T1641 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 400263 T1630 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 400265 T1630 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 400264 T1641 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solrj_collection_shard1_replica2/'
   [junit4]   2> 400263 T1598 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solrj_collection_shard2_replica2
   [junit4]   2> 400266 T1598 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 400266 T1630 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 400267 T1630 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solrj_collection_shard2_replica1/'
   [junit4]   2> 400269 T1598 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 400269 T1598 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 400271 T1598 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 400272 T1598 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solrj_collection_shard2_replica2/'
   [junit4]   2> 400325 T1616 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 400328 T1641 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 400330 T1630 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 400333 T1598 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 400389 T1616 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 400391 T1616 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 400394 T1641 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 400396 T1630 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 400396 T1641 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 400397 T1630 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 400397 T1598 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 400398 T1598 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 400400 T1616 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 400405 T1641 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 400406 T1630 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 400408 T1598 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 400777 T1641 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 400782 T1641 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 400785 T1641 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 400787 T1630 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 400794 T1630 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 400797 T1641 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 400798 T1630 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 400798 T1598 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 400799 T1641 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 400803 T1598 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 400803 T1641 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 400809 T1641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 400811 T1641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 400811 T1641 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 400812 T1598 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 400812 T1641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 400813 T1641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 400813 T1641 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 400813 T1641 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solrj_collection_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solrj_collection_shard1_replica2/data/
   [junit4]   2> 400814 T1641 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d3a8a09
   [junit4]   2> 400814 T1641 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solrj_collection_shard1_replica2/data
   [junit4]   2> 400815 T1641 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 400815 T1641 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 400815 T1641 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393305167011/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 400816 T1641 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2077693432, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 400817 T1641 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@33dab414 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@343eed11),segFN=segments_1,generation=1}
   [junit4]   2> 400817 T1641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 400818 T1630 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 400820 T1641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 400821 T1641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 400821 T1641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 400821 T1630 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 400821 T1641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 400822 T1641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 400822 T1641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 400823 T1641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 400823 T1641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 400823 T1641 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 400823 T1598 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 400823 T1641 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 400824 T1641 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 400824 T1641 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 400824 T1630 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 400825 T1641 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 400825 T1641 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 400825 T1630 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 400826 T1641 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 400826 T1630 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 400826 T1641 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 400826 T1598 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 400826 T1630 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 400827 T1641 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 400828 T1630 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 400828 T1630 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 400828 T1630 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 400828 T1630 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solrj_collection_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solrj_collection_shard2_replica1/data/
   [junit4]   2> 400828 T1630 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d3a8a09
   [junit4]   2> 400829 T1630 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solrj_collection_shard2_replica1/data
   [junit4]   2> 400829 T1641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 400829 T1630 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 400830 T1630 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 400830 T1598 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 400830 T1630 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393305165721/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 400831 T1630 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2077693432, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 400832 T1598 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 400832 T1630 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@53e8a84f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@555026dc),segFN=segments_1,generation=1}
   [junit4]   2> 400832 T1598 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 400832 T1641 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 400832 T1630 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 400833 T1641 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 400832 T1598 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 400833 T1641 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=88.0341796875, floorSegmentMB=0.66015625, forceMergeDeletesPctAllowed=8.633567282597562, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 400834 T1641 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@33dab414 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@343eed11),segFN=segments_1,generation=1}
   [junit4]   2> 400834 T1598 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 400834 T1641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 400834 T1598 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 400835 T1641 oass.SolrIndexSearcher.<init> Opening Searcher@5f7e47c8[solrj_collection_shard1_replica2] main
   [junit4]   2> 400835 T1598 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 400835 T1598 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solrj_collection_shard2_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solrj_collection_shard2_replica2/data/
   [junit4]   2> 400836 T1598 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d3a8a09
   [junit4]   2> 400836 T1616 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 400836 T1598 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solrj_collection_shard2_replica2/data
   [junit4]   2> 400836 T1630 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 400837 T1598 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 400837 T1630 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 400837 T1598 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 400837 T1630 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 400838 T1598 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393305164054/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 400838 T1630 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 400838 T1598 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2077693432, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 400838 T1630 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 400839 T1630 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 400839 T1598 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4d1d5468 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@232f1387),segFN=segments_1,generation=1}
   [junit4]   2> 400839 T1598 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 400839 T1668 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@5f7e47c8[solrj_collection_shard1_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 400839 T1641 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 400839 T1630 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 400840 T1641 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:http://127.0.0.1:42878/z_/rj collection:solrj_collection shard:shard1
   [junit4]   2> 400840 T1630 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 400840 T1630 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 400841 T1616 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 400841 T1630 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 400841 T1641 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 400841 T1630 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 400842 T1630 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 400842 T1630 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 400842 T1598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 400842 T1630 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 400842 T1598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 400843 T1630 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 400843 T1598 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 400843 T1616 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 400843 T1598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 400843 T1630 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 400844 T1598 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 400844 T1630 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 400845 T1598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 400845 T1598 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 400846 T1598 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 400846 T1630 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 400846 T1598 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 400846 T1598 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 400847 T1598 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 400847 T1598 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 400847 T1641 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 400848 T1598 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 400848 T1598 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 400849 T1598 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 400849 T1630 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 400849 T1630 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 400849 T1598 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 400849 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 400850 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 400850 T1641 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=-1
   [junit4]   2> 400850 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 400850 T1598 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 400850 T1630 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=88.0341796875, floorSegmentMB=0.66015625, forceMergeDeletesPctAllowed=8.633567282597562, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 400851 T1603 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 400850 T1641 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
   [junit4]   2> 400851 T1641 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 400851 T1630 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@53e8a84f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@555026dc),segFN=segments_1,generation=1}
   [junit4]   2> ASYNC  NEW_CORE C4016 name=solrj_collection_shard1_replica2 org.apache.solr.core.SolrCore@49955a43 url=http://127.0.0.1:42878/z_/rj/solrj_collection_shard1_replica2 node=127.0.0.1:42878_z_%2Frj C4016_STATE=coll:solrj_collection core:solrj_collection_shard1_replica2 props:{state=down, base_url=http://127.0.0.1:42878/z_/rj, core=solrj_collection_shard1_replica2, node_name=127.0.0.1:42878_z_%2Frj}
   [junit4]   2> 400851 T1641 C4016 P42878 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42878/z_/rj/solrj_collection_shard1_replica2/
   [junit4]   2> 400851 T1598 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 400852 T1641 C4016 P42878 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica2 url=http://127.0.0.1:42878/z_/rj START replicas=[http://127.0.0.1:53901/z_/rj/solrj_collection_shard1_replica1/] nUpdates=100
   [junit4]   2> 400852 T1630 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 400853 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 400853 T1641 C4016 P42878 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica2 url=http://127.0.0.1:42878/z_/rj DONE.  We have no versions.  sync failed.
   [junit4]   2> 400854 T1641 C4016 P42878 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 400853 T1630 oass.SolrIndexSearcher.<init> Opening Searcher@1a13edc6[solrj_collection_shard2_replica1] main
   [junit4]   2> 400854 T1641 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 400855 T1641 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42878/z_/rj/solrj_collection_shard1_replica2/ shard1
   [junit4]   2> 400855 T1641 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard1
   [junit4]   2> 400855 T1598 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 400856 T1598 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 400856 T1598 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=88.0341796875, floorSegmentMB=0.66015625, forceMergeDeletesPctAllowed=8.633567282597562, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 400857 T1616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 400857 T1598 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4d1d5468 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@232f1387),segFN=segments_1,generation=1}
   [junit4]   2> 400857 T1598 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 400857 T1670 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica1] Registered new searcher Searcher@1a13edc6[solrj_collection_shard2_replica1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 400857 T1598 oass.SolrIndexSearcher.<init> Opening Searcher@52e9b6e0[solrj_collection_shard2_replica2] main
   [junit4]   2> 400860 T1616 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 400860 T1672 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@52e9b6e0[solrj_collection_shard2_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 400860 T1630 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica1
   [junit4]   2> 400861 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 400861 T1630 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica1 address:http://127.0.0.1:37321/z_/rj collection:solrj_collection shard:shard2
   [junit4]   2> 400862 T1630 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 400863 T1598 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
   [junit4]   2> 400863 T1598 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica2 address:http://127.0.0.1:53141/z_/rj collection:solrj_collection shard:shard2
   [junit4]   2> 400863 T1616 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 400864 T1598 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 400864 T1616 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 400864 T1616 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 400864 T1602 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 400865 T1616 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 400866 T1616 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 400866 T1616 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 400866 T1616 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 400867 T1616 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/solrj_collection_shard1_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/solrj_collection_shard1_replica1/data/
   [junit4]   2> 400867 T1616 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d3a8a09
   [junit4]   2> 400867 T1616 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/solrj_collection_shard1_replica1/data
   [junit4]   2> 400868 T1630 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 400868 T1616 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393305164427/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 400868 T1616 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory './

[...truncated too long message...]

ative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 522314 T1662 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 522314 T1662 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 522314 T1662 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 522315 T1662 oasc.SolrCore.closeSearcher [nodes_used_collection_shard2_replica2] Closing main searcher on request.
   [junit4]   2> 522316 T1662 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 522316 T1662 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399/nodes_used_collection_shard2_replica2/data [CachedDir<<refCount=0;path=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399/nodes_used_collection_shard2_replica2/data;done=false>>]
   [junit4]   2> 522316 T1662 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399/nodes_used_collection_shard2_replica2/data
   [junit4]   2> 522316 T1662 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399/nodes_used_collection_shard2_replica2/data/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399/nodes_used_collection_shard2_replica2/data/index;done=false>>]
   [junit4]   2> 522317 T1662 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393305168399/nodes_used_collection_shard2_replica2/data/index
   [junit4]   2> 522317 T1662 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard2
   [junit4]   2> 522317 T1662 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:414)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 522318 T1662 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/nodes_used_collection/leader_elect/shard2/election/91311647210930190-core_node4-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 522320 T1662 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 522320 T1662 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene46: {multiDefault=PostingsFormat(name=MockRandom), timestamp=PostingsFormat(name=MockRandom), text=PostingsFormat(name=FSTPulsing41), range_facet_sl=FST41, intDefault=Pulsing41(freqCutoff=1 minBlockSize=26 maxBlockSize=99), a_t=PostingsFormat(name=MockRandom), a_si=Pulsing41(freqCutoff=1 minBlockSize=26 maxBlockSize=99), _version_=Pulsing41(freqCutoff=1 minBlockSize=26 maxBlockSize=99), range_facet_si=PostingsFormat(name=MockRandom), range_facet_l=PostingsFormat(name=MockRandom), other_tl1=Pulsing41(freqCutoff=1 minBlockSize=26 maxBlockSize=99), id=FST41}, docValues:{timestamp=DocValuesFormat(name=Lucene45)}, sim=DefaultSimilarity, locale=lv, timezone=America/Santarem
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=92691696,total=342884352
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrQueryParserDefaultOperatorResource, TestFastWriter, ChaosMonkeyNothingIsSafeTest, TestLazyCores, TestSolrXml, StandardRequestHandlerTest, AssignTest, TestAtomicUpdateErrorCases, PathHierarchyTokenizerFactoryTest, TestSolrJ, CustomCollectionTest, ShardRoutingCustomTest, DeleteInactiveReplicaTest, TestStressReorder, TestSolrXmlPersistence, TestSchemaResource, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 129.38s, 1 test, 1 failure <<< FAILURES!

[...truncated 821 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1606 tests, 1 failure, 35 ignored (7 assumptions)

Total time: 46 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/ibm-j9-jdk7) - Build # 9587 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9587/
Java: 64bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([6FC43740DCFE6DD8]:0)


REGRESSION:  org.apache.solr.core.TestLazyCores.testRace

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([6FC43740DCFE6DD8]:0)


FAILED:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([6FC43740DCFE6DD8:34F4CC88DFB9CEA4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)




Build Log:
[...truncated 10660 lines...]
   [junit4] Suite: org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest
   [junit4]   2> 553837 T2140 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 553838 T2140 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827
   [junit4]   2> 553838 T2140 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 553839 T2140 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 553839 T2140 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 553839 T2140 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 553870 T2140 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 553913 T2140 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 553914 T2140 oass.IndexSchema.readSchema Reading Solr Schema from schema12.xml
   [junit4]   2> 553919 T2140 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 554192 T2140 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 554194 T2140 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 554195 T2140 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 554200 T2140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 554203 T2140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 554206 T2140 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 554206 T2140 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 554207 T2140 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 554241 T2140 oasc.CoreContainer.<init> New CoreContainer -53997767
   [junit4]   2> 554242 T2140 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 554243 T2140 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 554243 T2140 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 554243 T2140 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 554244 T2140 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 554244 T2140 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 554244 T2140 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 554244 T2140 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 554245 T2140 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 554245 T2140 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 554248 T2140 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 554248 T2140 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 554248 T2140 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 554251 T2141 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 554251 T2141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 554252 T2141 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 554252 T2141 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 554252 T2141 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 554292 T2141 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 554335 T2141 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 554336 T2141 oass.IndexSchema.readSchema Reading Solr Schema from schema12.xml
   [junit4]   2> 554341 T2141 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 554626 T2141 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 554629 T2141 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 554630 T2141 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 554635 T2141 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 554638 T2141 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 554641 T2141 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 554641 T2141 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827/
   [junit4]   2> 554642 T2141 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@dbcedac0
   [junit4]   2> 554642 T2141 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827
   [junit4]   2> 554643 T2141 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827/index/
   [junit4]   2> 554643 T2141 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827/index' doesn't exist. Creating new index...
   [junit4]   2> 554643 T2141 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827/index
   [junit4]   2> 554644 T2141 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=69.0546875, floorSegmentMB=0.603515625, forceMergeDeletesPctAllowed=6.854698101993607, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4096309826881839
   [junit4]   2> 554644 T2141 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_1,generation=1}
   [junit4]   2> 554645 T2141 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 554645 T2141 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 554647 T2141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 554647 T2141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 554647 T2141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 554648 T2141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 554648 T2141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 554648 T2141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 554648 T2141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 554649 T2141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 554649 T2141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 554649 T2141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 554650 T2141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 554650 T2141 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 554650 T2141 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 554650 T2141 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 554651 T2141 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 554651 T2141 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 554651 T2141 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 554652 T2141 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 554652 T2141 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 554652 T2141 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 554653 T2141 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 554653 T2141 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 554653 T2141 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 554654 T2141 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 554654 T2141 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 554654 T2141 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 554655 T2141 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 554655 T2141 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 554655 T2141 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 554656 T2141 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 554656 T2141 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 554656 T2141 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 554657 T2141 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 554657 T2141 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 554657 T2141 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 554659 T2141 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 554660 T2141 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 554661 T2141 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 554661 T2141 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 554662 T2141 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=165862130, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 554662 T2141 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_1,generation=1}
   [junit4]   2> 554663 T2141 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 554663 T2141 oass.SolrIndexSearcher.<init> Opening Searcher@d686ff3b[collection1] main
   [junit4]   2> 554664 T2141 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 554765 T2141 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 554779 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d686ff3b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 554781 T2141 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 554784 T2140 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 554788 T2140 oas.SolrTestCaseJ4.setUp ###Starting testNonStringFieldsValues
   [junit4]   2> ASYNC  NEW_CORE C986 name=collection1 org.apache.solr.core.SolrCore@e7b2e11d
   [junit4]   2> 554789 T2140 C986 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 554790 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 554795 T2140 C986 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 554796 T2140 C986 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_2,generation=2}
   [junit4]   2> 554797 T2140 C986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 554798 T2140 C986 oass.SolrIndexSearcher.<init> Opening Searcher@59c4f6eb[collection1] main
   [junit4]   2> 554799 T2140 C986 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 554802 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@59c4f6eb[collection1] main{StandardDirectoryReader(segments_2:2:nrt)}
   [junit4]   2> 554803 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 8
   [junit4]   2> 554807 T2140 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 554950 T2140 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 554952 T2140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_3,generation=3}
   [junit4]   2> 554952 T2140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 554953 T2140 oass.SolrIndexSearcher.<init> Opening Searcher@899ae9c[collection1] main
   [junit4]   2> 554954 T2140 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 554956 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@899ae9c[collection1] main{StandardDirectoryReader(segments_3:4:nrt _0(5.0):C5/1:delGen=1)}
   [junit4]   2> 554956 T2140 oas.SolrTestCaseJ4.tearDown ###Ending testNonStringFieldsValues
   [junit4]   2> 554962 T2140 oas.SolrTestCaseJ4.setUp ###Starting testNonIndexedSignatureField
   [junit4]   2> 554962 T2140 C986 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 554963 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 554964 T2140 C986 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 554965 T2140 C986 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_3,generation=3}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_4,generation=4}
   [junit4]   2> 554965 T2140 C986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
   [junit4]   2> 554966 T2140 C986 oass.SolrIndexSearcher.<init> Opening Searcher@2b7c3ea1[collection1] main
   [junit4]   2> 554966 T2140 C986 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 554967 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2b7c3ea1[collection1] main{StandardDirectoryReader(segments_4:5:nrt)}
   [junit4]   2> 554968 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 554972 T2140 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 554973 T2140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_4,generation=4}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_5,generation=5}
   [junit4]   2> 554974 T2140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
   [junit4]   2> 554975 T2140 oass.SolrIndexSearcher.<init> Opening Searcher@6b6d6b64[collection1] main
   [junit4]   2> 554975 T2140 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 554976 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b6d6b64[collection1] main{StandardDirectoryReader(segments_5:7:nrt _1(5.0):C2)}
   [junit4]   2> 554977 T2140 oas.SolrTestCaseJ4.tearDown ###Ending testNonIndexedSignatureField
   [junit4]   2> 554981 T2140 oas.SolrTestCaseJ4.setUp ###Starting testDupeAllFieldsDetection
   [junit4]   2> 554981 T2140 C986 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 554982 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 554983 T2140 C986 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 554983 T2140 C986 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_5,generation=5}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_6,generation=6}
   [junit4]   2> 554984 T2140 C986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
   [junit4]   2> 554985 T2140 C986 oass.SolrIndexSearcher.<init> Opening Searcher@c8f4969a[collection1] main
   [junit4]   2> 554985 T2140 C986 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 554987 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c8f4969a[collection1] main{StandardDirectoryReader(segments_6:8:nrt)}
   [junit4]   2> 554987 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 554992 T2140 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 554993 T2140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_6,generation=6}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_7,generation=7}
   [junit4]   2> 554994 T2140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
   [junit4]   2> 554995 T2140 oass.SolrIndexSearcher.<init> Opening Searcher@cbc4adea[collection1] main
   [junit4]   2> 554996 T2140 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 554997 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@cbc4adea[collection1] main{StandardDirectoryReader(segments_7:10:nrt _2(5.0):C3)}
   [junit4]   2> 554997 T2140 oas.SolrTestCaseJ4.tearDown ###Ending testDupeAllFieldsDetection
   [junit4]   2> 555001 T2140 oas.SolrTestCaseJ4.setUp ###Starting testFailNonIndexedSigWithOverwriteDupes
   [junit4]   2> 555001 T2140 C986 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 555002 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 555003 T2140 C986 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555004 T2140 C986 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_7,generation=7}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_8,generation=8}
   [junit4]   2> 555004 T2140 C986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 8
   [junit4]   2> 555005 T2140 C986 oass.SolrIndexSearcher.<init> Opening Searcher@df1b4306[collection1] main
   [junit4]   2> 555005 T2140 C986 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555006 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@df1b4306[collection1] main{StandardDirectoryReader(segments_8:11:nrt)}
   [junit4]   2> 555007 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 555008 T2140 oas.SolrTestCaseJ4.tearDown ###Ending testFailNonIndexedSigWithOverwriteDupes
   [junit4]   2> 555012 T2140 oas.SolrTestCaseJ4.setUp ###Starting testMultiThreaded
   [junit4]   2> 555012 T2140 C986 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 555013 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 555014 T2140 C986 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555014 T2140 C986 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_8,generation=8}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_9,generation=9}
   [junit4]   2> 555015 T2140 C986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 9
   [junit4]   2> 555015 T2140 C986 oass.SolrIndexSearcher.<init> Opening Searcher@e3d853b8[collection1] main
   [junit4]   2> 555016 T2140 C986 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555017 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e3d853b8[collection1] main{StandardDirectoryReader(segments_9:12:nrt)}
   [junit4]   2> 555018 T2140 C986 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 555026 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555134 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_9,generation=9}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_a,generation=10}
   [junit4]   2> 555135 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 10
   [junit4]   2> 555160 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@173a1635[collection1] main
   [junit4]   2> 555161 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555161 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555161 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_a,generation=10}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_b,generation=11}
   [junit4]   2> 555162 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 11
   [junit4]   2> 555162 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@173a1635[collection1] main{StandardDirectoryReader(segments_a:34:nrt _b(5.0):C14/12:delGen=2)}
   [junit4]   2> 555163 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@d1039253[collection1] main
   [junit4]   2> 555163 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555163 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555164 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555164 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d1039253[collection1] main{StandardDirectoryReader(segments_a:34:nrt _b(5.0):C14/12:delGen=2)}
   [junit4]   2> 555166 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@3b5b00e8[collection1] main
   [junit4]   2> 555167 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555167 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555168 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b5b00e8[collection1] main{StandardDirectoryReader(segments_b:36:nrt _b(5.0):C14/12:delGen=2 _c(5.0):C1)}
   [junit4]   2> 555169 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_b,generation=11}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_c,generation=12}
   [junit4]   2> 555170 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 12
   [junit4]   2> 555172 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@c0cbb997[collection1] main
   [junit4]   2> 555173 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555173 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555174 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c0cbb997[collection1] main{StandardDirectoryReader(segments_c:42:nrt _b(5.0):C14/13:delGen=3 _d(5.0):C1 _e(5.0):C1)}
   [junit4]   2> 555174 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_c,generation=12}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_d,generation=13}
   [junit4]   2> 555174 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 13
   [junit4]   2> 555175 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@445a8da6[collection1] main
   [junit4]   2> 555175 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555176 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555176 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555177 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@445a8da6[collection1] main{StandardDirectoryReader(segments_d:44:nrt _b(5.0):C14/13:delGen=3 _e(5.0):C1)}
   [junit4]   2> 555179 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@68224dd3[collection1] main
   [junit4]   2> 555179 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555180 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555180 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@68224dd3[collection1] main{StandardDirectoryReader(segments_d:48:nrt _e(5.0):C1 _f(5.0):C1)}
   [junit4]   2> 555182 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_d,generation=13}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_e,generation=14}
   [junit4]   2> 555183 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 14
   [junit4]   2> 555185 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@78c6b26e[collection1] main
   [junit4]   2> 555186 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555187 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555187 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78c6b26e[collection1] main{StandardDirectoryReader(segments_e:56:nrt _e(5.0):C1 _h(5.0):C1)}
   [junit4]   2> 555187 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_e,generation=14}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_f,generation=15}
   [junit4]   2> 555188 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 15
   [junit4]   2> 555188 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@f7679671[collection1] main
   [junit4]   2> 555189 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555189 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555190 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555190 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f7679671[collection1] main{StandardDirectoryReader(segments_e:56:nrt _e(5.0):C1 _h(5.0):C1)}
   [junit4]   2> 555192 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@50cf0c8d[collection1] main
   [junit4]   2> 555192 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555193 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555193 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@50cf0c8d[collection1] main{StandardDirectoryReader(segments_f:58:nrt _e(5.0):C1 _h(5.0):C1 _i(5.0):C1)}
   [junit4]   2> 555195 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_f,generation=15}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_g,generation=16}
   [junit4]   2> 555196 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 16
   [junit4]   2> 555198 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@45bcb3aa[collection1] main
   [junit4]   2> 555199 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555199 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555200 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@45bcb3aa[collection1] main{StandardDirectoryReader(segments_g:66:nrt _e(5.0):C1 _k(5.0):C1)}
   [junit4]   2> 555200 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_g,generation=16}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_h,generation=17}
   [junit4]   2> 555200 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 17
   [junit4]   2> 555201 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@90b7f849[collection1] main
   [junit4]   2> 555201 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555202 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555202 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555203 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@90b7f849[collection1] main{StandardDirectoryReader(segments_g:66:nrt _e(5.0):C1 _k(5.0):C1)}
   [junit4]   2> 555204 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@6bba0355[collection1] main
   [junit4]   2> 555205 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555205 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555206 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6bba0355[collection1] main{StandardDirectoryReader(segments_h:71:nrt _k(5.0):C1 _l(5.0):C1)}
   [junit4]   2> 555207 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_h,generation=17}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_i,generation=18}
   [junit4]   2> 555208 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 18
   [junit4]   2> 555210 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@20651cb0[collection1] main
   [junit4]   2> 555211 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555212 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555212 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@20651cb0[collection1] main{StandardDirectoryReader(segments_i:78:nrt _n(5.0):C1)}
   [junit4]   2> 555212 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_i,generation=18}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_j,generation=19}
   [junit4]   2> 555213 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 19
   [junit4]   2> 555213 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@839d7f5[collection1] main
   [junit4]   2> 555213 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555214 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555214 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555215 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@839d7f5[collection1] main{StandardDirectoryReader(segments_i:78:nrt _n(5.0):C1)}
   [junit4]   2> 555216 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@6a63aa27[collection1] main
   [junit4]   2> 555217 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555217 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555218 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6a63aa27[collection1] main{StandardDirectoryReader(segments_j:80:nrt _n(5.0):C1 _o(5.0):C1)}
   [junit4]   2> 555220 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_j,generation=19}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_k,generation=20}
   [junit4]   2> 555220 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 20
   [junit4]   2> 555222 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@e8da0bfa[collection1] main
   [junit4]   2> 555223 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555224 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555224 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e8da0bfa[collection1] main{StandardDirectoryReader(segments_k:88:nrt _n(5.0):C1 _q(5.0):C1)}
   [junit4]   2> 555224 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_k,generation=20}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_l,generation=21}
   [junit4]   2> 555225 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 21
   [junit4]   2> 555225 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@8c3ddc44[collection1] main
   [junit4]   2> 555225 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555226 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555226 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555227 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8c3ddc44[collection1] main{StandardDirectoryReader(segments_k:88:nrt _n(5.0):C1 _q(5.0):C1)}
   [junit4]   2> 555228 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@555d83dd[collection1] main
   [junit4]   2> 555229 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555229 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555230 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@555d83dd[collection1] main{StandardDirectoryReader(segments_l:90:nrt _n(5.0):C1 _q(5.0):C1 _r(5.0):C1)}
   [junit4]   2> 555232 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_l,generation=21}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_m,generation=22}
   [junit4]   2> 555232 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 22
   [junit4]   2> 555234 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@88b93ee[collection1] main
   [junit4]   2> 555235 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555236 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555236 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@88b93ee[collection1] main{StandardDirectoryReader(segments_m:98:nrt _n(5.0):C1 _q(5.0):C1 _t(5.0):C1)}
   [junit4]   2> 555236 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_m,generation=22}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_n,generation=23}
   [junit4]   2> 555237 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 23
   [junit4]   2> 555237 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@dc1fc9fb[collection1] main
   [junit4]   2> 555238 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555238 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555239 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555239 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@dc1fc9fb[collection1] main{StandardDirectoryReader(segments_m:98:nrt _n(5.0):C1 _q(5.0):C1 _t(5.0):C1)}
   [junit4]   2> 555241 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@7a45235[collection1] main
   [junit4]   2> 555241 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555242 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555242 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a45235[collection1] main{StandardDirectoryReader(segments_n:103:nrt _n(5.0):C1 _q(5.0):C1 _u(5.0):C1)}
   [junit4]   2> 555244 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_n,generation=23}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_o,generation=24}
   [junit4]   2> 555245 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 24
   [junit4]   2> 555247 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@26700a6e[collection1] main
   [junit4]   2> 555248 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555248 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555249 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_o,generation=24}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_p,generation=25}
   [junit4]   2> 555249 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 25
   [junit4]   2> 555249 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@26700a6e[collection1] main{StandardDirectoryReader(segments_o:110:nrt _n(5.0):C1 _w(5.0):C1)}
   [junit4]   2> 555250 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@80e7b5a[collection1] main
   [junit4]   2> 555250 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555251 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555251 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555252 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@80e7b5a[collection1] main{StandardDirectoryReader(segments_o:110:nrt _n(5.0):C1 _w(5.0):C1)}
   [junit4]   2> 555254 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@1d45a77b[collection1] main
   [junit4]   2> 555254 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555255 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555256 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d45a77b[collection1] main{StandardDirectoryReader(segments_p:114:nrt _w(5.0):C1 _x(5.0):C1)}
   [junit4]   2> 555257 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_p,generation=25}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_q,generation=26}
   [junit4]   2> 555257 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 26
   [junit4]   2> 555258 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@ee69d9b0[collection1] main
   [junit4]   2> 555259 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555259 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555260 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ee69d9b0[collection1] main{StandardDirectoryReader(segments_q:118:nrt _y(5.0):C1)}
   [junit4]   2> 555261 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_q,generation=26}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_r,generation=27}
   [junit4]   2> 555261 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 27
   [junit4]   2> 555261 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@48fbe2de[collection1] main
   [junit4]   2> 555262 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555262 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555263 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555263 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@48fbe2de[collection1] main{StandardDirectoryReader(segments_r:122:nrt _z(5.0):C1)}
   [junit4]   2> 555265 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@7367926a[collection1] main
   [junit4]   2> 555266 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555266 T2153 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555267 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7367926a[collection1] main{StandardDirectoryReader(segments_r:124:nrt _z(5.0):C1 _10(5.0):C1)}
   [junit4]   2> 555269 T2153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_r,generation=27}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_s,generation=28}
   [junit4]   2> 555269 T2153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 28
   [junit4]   2> 555270 T2153 oass.SolrIndexSearcher.<init> Opening Searcher@da1c8adf[collection1] main
   [junit4]   2> 555271 T2153 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555271 T2152 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555272 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@da1c8adf[collection1] main{StandardDirectoryReader(segments_s:128:nrt _11(5.0):C1)}
   [junit4]   2> 555273 T2152 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_s,generation=28}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_t,generation=29}
   [junit4]   2> 555273 T2152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 29
   [junit4]   2> 555273 T2152 oass.SolrIndexSearcher.<init> Opening Searcher@ff35a2c4[collection1] main
   [junit4]   2> 555274 T2152 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555274 T2151 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555275 T2151 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555275 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ff35a2c4[collection1] main{StandardDirectoryReader(segments_t:132:nrt _12(5.0):C1)}
   [junit4]   2> 555276 T2151 oass.SolrIndexSearcher.<init> Opening Searcher@226fff79[collection1] main
   [junit4]   2> 555276 T2151 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555277 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@226fff79[collection1] main{StandardDirectoryReader(segments_t:132:nrt _12(5.0):C1)}
   [junit4]   2> ASYNC  NEW_CORE C987 name=collection1 org.apache.solr.core.SolrCore@e7b2e11d
   [junit4]   2> 555278 T2140 C987 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555279 T2140 C987 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 555280 T2140 C987 oass.SolrIndexSearcher.<init> Opening Searcher@9826e2cf[collection1] main
   [junit4]   2> 555281 T2140 C987 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555282 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9826e2cf[collection1] main{StandardDirectoryReader(segments_t:132:nrt _12(5.0):C1)}
   [junit4]   2> 555282 T2140 C987 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 555284 T2140 oas.SolrTestCaseJ4.tearDown ###Ending testMultiThreaded
   [junit4]   2> 555290 T2140 oas.SolrTestCaseJ4.setUp ###Starting testDupeDetection
   [junit4]   2> 555290 T2140 C987 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 555291 T2140 C987 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 555293 T2140 C987 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555294 T2140 C987 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_t,generation=29}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_u,generation=30}
   [junit4]   2> 555295 T2140 C987 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 30
   [junit4]   2> 555296 T2140 C987 oass.SolrIndexSearcher.<init> Opening Searcher@a03faa86[collection1] main
   [junit4]   2> 555297 T2140 C987 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555298 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a03faa86[collection1] main{StandardDirectoryReader(segments_u:133:nrt)}
   [junit4]   2> 555298 T2140 C987 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 5
   [junit4]   2> 555303 T2140 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555305 T2140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_u,generation=30}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_v,generation=31}
   [junit4]   2> 555306 T2140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 31
   [junit4]   2> 555306 T2140 oass.SolrIndexSearcher.<init> Opening Searcher@a8da2801[collection1] main
   [junit4]   2> 555307 T2140 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555308 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a8da2801[collection1] main{StandardDirectoryReader(segments_v:135:nrt _13(5.0):C2)}
   [junit4]   2> 555311 T2140 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 555312 T2140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_v,generation=31}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1d457da6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e4149c3)),segFN=segments_w,generation=32}
   [junit4]   2> 555313 T2140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 32
   [junit4]   2> 555314 T2140 oass.SolrIndexSearcher.<init> Opening Searcher@ad2d0f21[collection1] main
   [junit4]   2> 555314 T2140 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 555315 T2142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ad2d0f21[collection1] main{StandardDirectoryReader(segments_w:138:nrt _13(5.0):C2/1:delGen=1 _14(5.0):C1)}
   [junit4]   2> 555316 T2140 oas.SolrTestCaseJ4.tearDown ###Ending testDupeDetection
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SignatureUpdateProcessorFactoryTest -Dtests.method=testDupeDetection -Dtests.seed=6FC43740DCFE6DD8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh_SG -Dtests.timezone=Asia/Dili -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.03s J1 | SignatureUpdateProcessorFactoryTest.testDupeDetection <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1> but was:<2>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6FC43740DCFE6DD8:34F4CC88DFB9CEA4]:0)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 555320 T2140 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 555321 T2140 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-53997767
   [junit4]   2> 555321 T2140 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@e7b2e11d
   [junit4]   2> 555329 T2140 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=42,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=253,cumulative_deletesById=0,cumulative_deletesByQuery=6,cumulative_errors=0}
   [junit4]   2> 555330 T2140 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 555330 T2140 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 555330 T2140 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 555331 T2140 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 555331 T2140 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 555332 T2140 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827;done=false>>]
   [junit4]   2> 555332 T2140 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827
   [junit4]   2> 555332 T2140 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827/index;done=false>>]
   [junit4]   2> 555332 T2140 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393293554827/index
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=DefaultSimilarity, locale=zh_SG, timezone=Asia/Dili
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/IBM Corporation 1.7.0 (64-bit)/cpus=8,threads=1,free=60630448,total=196280320
   [junit4]   2> NOTE: All tests run in this JVM: [CurrencyFieldOpenExchangeTest, TestOmitPositions, FieldAnalysisRequestHandlerTest, MultiTermTest, DistanceFunctionTest, TestCustomSort, TestJmxIntegration, TimeZoneUtilsTest, TestFastLRUCache, TestReload, TestDistributedGrouping, DocValuesMissingTest, TestQuerySenderListener, TestStressRecovery, TestFieldCollectionResource, TestUtils, XsltUpdateRequestHandlerTest, ShardSplitTest, TestInfoStreamLogging, TestSchemaVersionResource, DistribCursorPagingTest, DocValuesMultiTest, TestAddFieldRealTimeGet, TestPartialUpdateDeduplication, CachingDirectoryFactoryTest, TestRTGBase, TestIndexingPerformance, DistributedDebugComponentTest, TestIndexSearcher, TestSolrQueryParserResource, TestCSVResponseWriter, DirectSolrSpellCheckerTest, TestBinaryResponseWriter, DirectUpdateHandlerTest, NumericFieldsTest, DistributedSuggestComponentTest, RAMDirectoryFactoryTest, TestArbitraryIndexDir, TestNumberUtils, OpenCloseCoreStressTest, AliasIntegrationTest, TestBinaryField, TestFieldTypeCollectionResource, TestSolrJ, PreAnalyzedUpdateProcessorTest, SolrCoreCheckLockOnStartupTest, AnalysisErrorHandlingTest, FileUtilsTest, DefaultValueUpdateProcessorTest, ZkSolrClientTest, IndexBasedSpellCheckerTest, TestSolrXml, StandardRequestHandlerTest, TestShardHandlerFactory, TestSweetSpotSimilarityFactory, TestCopyFieldCollectionResource, HardAutoCommitTest, TestZkChroot, AlternateDirectoryTest, HdfsBasicDistributedZk2Test, SuggesterWFSTTest, TestFastOutputStream, TestSchemaResource, TestRangeQuery, TestDistribDocBasedVersion, TestWordDelimiterFilterFactory, TestRecoveryHdfs, SolrTestCaseJ4Test, PingRequestHandlerTest, TestSchemaNameResource, SuggestComponentTest, TestUpdate, TestDFRSimilarityFactory, TestCoreDiscovery, CurrencyFieldXmlFileTest, SpellCheckComponentTest, PathHierarchyTokenizerFactoryTest, TestCSVLoader, LeaderElectionIntegrationTest, TriLevelCompositeIdRoutingTest, CollectionsAPIDistributedZkTest, ConvertedLegacyTest, TestRecovery, TestCodecSupport, CoreMergeIndexesAdminHandlerTest, DirectSolrConnectionTest, TestAtomicUpdateErrorCases, SpellingQueryConverterTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, SyncSliceTest, OverseerTest, BasicZkTest, LeaderElectionTest, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, TestFaceting, TestHashPartitioner, TestRealTimeGet, TestMultiCoreConfBootstrap, TestStressVersions, StatsComponentTest, TestGroupingSearch, TestFiltering, TestBadConfig, OverseerCollectionProcessorTest, HighlighterTest, SoftAutoCommitTest, ShowFileRequestHandlerTest, SolrIndexSplitterTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest]
   [junit4] Completed on J1 in 1.51s, 6 tests, 1 failure <<< FAILURES!

[...truncated 546 lines...]
   [junit4] Suite: org.apache.solr.core.TestLazyCores
   [junit4]   2> 502802 T2047 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 502802 T2047 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254
   [junit4]   2> 502803 T2047 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 502804 T2047 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 502804 T2047 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 502804 T2047 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 502837 T2047 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 502869 T2047 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 502870 T2047 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 502874 T2047 oass.IndexSchema.readSchema [null] Schema name=tiny
   [junit4]   2> 502888 T2047 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 502889 T2047 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 502889 T2047 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 502890 T2047 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 502928 T2047 oasc.CoreContainer.<init> New CoreContainer -1637354867
   [junit4]   2> 502928 T2047 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 502929 T2047 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 502930 T2047 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 502930 T2047 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 502930 T2047 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 502931 T2047 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 502931 T2047 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 502932 T2047 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 502932 T2047 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 502932 T2047 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 502936 T2047 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 502936 T2047 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 502936 T2047 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 502939 T2048 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 502940 T2048 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 502940 T2048 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 502941 T2048 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 502941 T2048 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 502972 T2048 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 503002 T2048 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 503003 T2048 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 503006 T2048 oass.IndexSchema.readSchema [collection1] Schema name=tiny
   [junit4]   2> 503017 T2048 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 503018 T2048 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 503019 T2048 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254/
   [junit4]   2> 503019 T2048 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 503020 T2048 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254
   [junit4]   2> 503020 T2048 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254/index/
   [junit4]   2> 503020 T2048 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254/index' doesn't exist. Creating new index...
   [junit4]   2> 503021 T2048 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254/index
   [junit4]   2> 503021 T2048 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 503021 T2048 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a2276b38 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@849a8cf7),segFN=segments_1,generation=1}
   [junit4]   2> 503022 T2048 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 503023 T2048 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 503023 T2048 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 503023 T2048 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 503023 T2048 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 503025 T2048 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 503026 T2048 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 503027 T2048 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 503027 T2048 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2774170897765993]
   [junit4]   2> 503028 T2048 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a2276b38 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@849a8cf7),segFN=segments_1,generation=1}
   [junit4]   2> 503028 T2048 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 503028 T2048 oass.SolrIndexSearcher.<init> Opening Searcher@340b2593[collection1] main
   [junit4]   2> 503029 T2048 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 503029 T2049 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@340b2593[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 503031 T2047 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 503034 T2047 oas.SolrTestCaseJ4.setUp ###Starting testPersistence
   [junit4]   2> 503047 T2047 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/'
   [junit4]   2> 503053 T2047 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/solr.xml
   [junit4]   2> 503090 T2047 oasc.CoreContainer.<init> New CoreContainer -2085358489
   [junit4]   2> 503091 T2047 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/]
   [junit4]   2> 503092 T2047 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 503092 T2047 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 503093 T2047 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 503093 T2047 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 503094 T2047 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 503094 T2047 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 503094 T2047 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 503095 T2047 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 503095 T2047 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 503098 T2047 oasc.SolrCores.allocateLazyCores Allocating transient cache for 4 transient cores
   [junit4]   2> 503099 T2047 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 503099 T2047 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 503099 T2047 oasc.CoreContainer.load Host Name: null
   [junit4]   2> 503109 T2051 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection1
   [junit4]   2> 503109 T2053 oasc.CoreContainer.create Creating SolrCore 'collectionLazy5' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection5
   [junit4]   2> 503109 T2051 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection1/'
   [junit4]   2> 503109 T2052 oasc.CoreContainer.create Creating SolrCore 'collectionLazy2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection2
   [junit4]   2> 503110 T2053 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection5/'
   [junit4]   2> 503110 T2052 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection2/'
   [junit4]   2> 503143 T2051 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 503149 T2053 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 503151 T2052 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 503175 T2051 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 503175 T2051 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 503179 T2051 oass.IndexSchema.readSchema [collection1] Schema name=tiny
   [junit4]   2> 503191 T2051 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 503192 T2051 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 503192 T2051 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection1/data/
   [junit4]   2> 503192 T2051 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 503193 T2052 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 503193 T2051 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection1/data
   [junit4]   2> 503193 T2051 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection1/data/index/
   [junit4]   2> 503193 T2052 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 503193 T2053 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 503194 T2051 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk

[...truncated too long message...]

;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection2/data;done=false>>]
   [junit4]   2> 7722930 T2047 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection2/data
   [junit4]   2> 7722931 T2047 oasc.SolrCore.close [collectionLazy3]  CLOSING SolrCore org.apache.solr.core.SolrCore@9e72dd9
   [junit4]   2> 7722931 T2047 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 7722932 T2047 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 7722932 T2047 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 7722932 T2047 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 7722933 T2047 oasc.SolrCore.closeSearcher [collectionLazy3] Closing main searcher on request.
   [junit4]   2> 7722933 T2047 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 7722934 T2047 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection3/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection3/data;done=false>>]
   [junit4]   2> 7722934 T2047 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection3/data
   [junit4]   2> 7722934 T2047 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection3/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection3/data/index;done=false>>]
   [junit4]   2> 7722935 T2047 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./TestLazyCores/collection3/data/index
   [junit4]   2> 7722942 T2047 oas.SolrTestCaseJ4.tearDown ###Ending testRace
   [junit4]   2> 7722946 T2047 oas.SolrTestCaseJ4.setUp ###Starting testCreateSame
   [junit4]   2> 7722948 T2047 oas.SolrTestCaseJ4.tearDown ###Ending testCreateSame
   [junit4]   2> 7722949 T2047 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 7722949 T2047 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1637354867
   [junit4]   2> 7722949 T2047 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7b6581ba
   [junit4]   2> 7722950 T2047 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 7722950 T2047 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 7722950 T2047 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 7722951 T2047 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 7722951 T2047 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 7722951 T2047 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 7722952 T2047 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254/index;done=false>>]
   [junit4]   2> 7722952 T2047 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254/index
   [junit4]   2> 7722952 T2047 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254;done=false>>]
   [junit4]   2> 7722953 T2047 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-TestLazyCores-1393293504254
   [junit4]   2> 7220153 T2046 ccr.ThreadLeakControl.tryToInterruptAll All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=en_NZ, timezone=Pacific/Palau
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/IBM Corporation 1.7.0 (64-bit)/cpus=8,threads=1,free=123246552,total=290979840
   [junit4]   2> NOTE: All tests run in this JVM: [DateMathParserTest, TestUniqueKeyFieldResource, CacheHeaderTest, TermsComponentTest, UnloadDistributedZkTest, DistributedSpellCheckComponentTest, URLClassifyProcessorTest, TestClassNameShortening, EnumFieldTest, TestSurroundQueryParser, TestLRUCache, EchoParamsTest, PrimitiveFieldTypeTest, TermVectorComponentTest, AutoCommitTest, CopyFieldTest, SpellPossibilityIteratorTest, MBeansHandlerTest, TestFreeTextSuggestions, TestHighlightDedupGrouping, CoreAdminHandlerTest, CursorMarkTest, TestTrie, ScriptEngineTest, TestPseudoReturnFields, HdfsSyncSliceTest, QueryParsingTest, HdfsLockFactoryTest, TestAnalyzeInfixSuggestions, TestModifyConfFiles, TestFieldResource, TestDocBasedVersionConstraints, SystemInfoHandlerTest, SolrXmlInZkTest, TestOverriddenPrefixQueryForCustomFieldType, DOMUtilTest, TestCharFilters, TestLMJelinekMercerSimilarityFactory, TestSolrXMLSerializer, ResponseLogComponentTest, RegexBoostProcessorTest, BadComponentTest, BlockDirectoryTest, TestNoOpRegenerator, SOLR749Test, TestLFUCache, TestWriterPerf, SliceStateTest, UpdateParamsTest, TestValueSourceCache, TestDocumentBuilder, TestRandomDVFaceting, TestDefaultSearchFieldResource, BinaryUpdateRequestHandlerTest, TestReplicationHandler, TestSchemaSimilarityResource, ParsingFieldUpdateProcessorsTest, TestPostingsSolrHighlighter, TestDocSet, TestElisionMultitermQuery, SearchHandlerTest, TestPHPSerializedResponseWriter, SliceStateUpdateTest, TestManagedSchema, TestQueryUtils, FastVectorHighlighterTest, CircularListTest, TestFieldTypeResource, TestDynamicFieldResource, InfoHandlerTest, QueryEqualityTest, ClusterStateTest, TestJoin, WordBreakSolrSpellCheckerTest, AbstractAnalyticsFacetTest, TestDistributedMissingSort, LukeRequestHandlerTest, BasicDistributedZkTest, ModifyConfFileTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestStandardQParsers, TestFunctionQuery, DistributedQueryElevationComponentTest, TestSystemIdResolver, TestCoreContainer, StressHdfsTest, TestSolrXmlPersistor, ShardRoutingTest, ReturnFieldsTest, DeleteReplicaTest, TestSolrDeletionPolicy1, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, TermVectorComponentDistributedTest, ZkControllerTest, TestStressReorder, DistributedTermsComponentTest, SimpleFacetsTest, TestSolr4Spatial, SolrCoreTest, SolrCmdDistributorTest, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, TestSort, BasicFunctionalityTest, TestLazyCores]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLazyCores -Dtests.seed=6FC43740DCFE6DD8 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_NZ -Dtests.timezone=Pacific/Palau -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestLazyCores (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6FC43740DCFE6DD8]:0)
   [junit4] Completed on J0 in 7220.17s, 7 tests, 2 errors <<< FAILURES!

[...truncated 12 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:458: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1605 tests, 1 suite-level error, 1 error, 1 failure, 74 ignored (7 assumptions)

Total time: 152 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9586 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9586/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseG1GC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([5135ECA5636D93E7]:0)


REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([5135ECA5636D93E7]:0)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Could not find collection:addReplicaColl

Stack Trace:
java.lang.AssertionError: Could not find collection:addReplicaColl
	at __randomizedtesting.SeedInfo.seed([5135ECA5636D93E7:D0D362BD1432F3DB]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:147)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:724)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1193)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10268 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 692098 T2975 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 692098 T2975 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 692102 T2975 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1393281955671
   [junit4]   2> 692103 T2975 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 692103 T2976 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 692203 T2975 oasc.ZkTestServer.run start zk server on port:35350
   [junit4]   2> 692204 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 692355 T2982 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19ae7b6 name:ZooKeeperConnection Watcher:127.0.0.1:35350 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 692356 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 692356 T2975 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 692363 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 692365 T2984 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19d0255 name:ZooKeeperConnection Watcher:127.0.0.1:35350/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 692365 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 692365 T2975 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 692368 T2975 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 692370 T2975 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 692372 T2975 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 692373 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 692374 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 692377 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 692377 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 692380 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 692381 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 692384 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 692385 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 692387 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 692388 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 692390 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 692390 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 692392 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 692393 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 692395 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 692395 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 692397 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 692398 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 692400 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 692400 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 692402 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 692403 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 692406 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 692407 T2986 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11cf783 name:ZooKeeperConnection Watcher:127.0.0.1:35350/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 692407 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 692408 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 692409 T2988 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13e446e name:ZooKeeperConnection Watcher:127.0.0.1:35350 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 692410 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 692410 T2975 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 692412 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 692413 T2990 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d63ab name:ZooKeeperConnection Watcher:127.0.0.1:35350/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 692413 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 692414 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 692414 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 692418 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 692418 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 692421 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 692421 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 692424 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 692424 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 692426 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 692426 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 692428 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 692429 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 692430 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 692431 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 692433 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 692433 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 692435 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 692436 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 692437 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 692438 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 692440 T2975 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 692440 T2975 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 692645 T2975 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 692648 T2975 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 692651 T2975 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57878
   [junit4]   2> 692652 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 692653 T2975 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 692653 T2975 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393281956011
   [junit4]   2> 692653 T2975 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393281956011/'
   [junit4]   2> 692663 T2975 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393281956011/solr.xml
   [junit4]   2> 692697 T2975 oasc.CoreContainer.<init> New CoreContainer 20928047
   [junit4]   2> 692698 T2975 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393281956011/]
   [junit4]   2> 692699 T2975 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 692699 T2975 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 692699 T2975 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 692700 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 692700 T2975 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 692700 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 692701 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 692701 T2975 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 692701 T2975 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 692705 T2975 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 692705 T2975 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 692706 T2975 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 692706 T2975 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35350/solr
   [junit4]   2> 692706 T2975 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 692707 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 692709 T3001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65ccb4 name:ZooKeeperConnection Watcher:127.0.0.1:35350 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 692709 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 692712 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 692713 T3003 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@aa6bf7 name:ZooKeeperConnection Watcher:127.0.0.1:35350/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 692713 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 692714 T2975 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 692718 T2975 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 692721 T2975 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 692723 T2975 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57878_
   [junit4]   2> 692724 T2975 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57878_
   [junit4]   2> 692727 T2975 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 692729 T2975 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 692733 T2975 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57878_
   [junit4]   2> 692733 T2975 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 692735 T2975 oasc.Overseer.start Overseer (id=91310126246985734-127.0.0.1:57878_-n_0000000000) starting
   [junit4]   2> 692737 T2975 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 692741 T3005 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 692742 T2975 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 692742 T3005 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 692745 T2975 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 692747 T2975 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 692749 T3004 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 692752 T2975 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 692752 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 692755 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 692756 T3008 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4840a8 name:ZooKeeperConnection Watcher:127.0.0.1:35350/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 692756 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 692757 T2975 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 692759 T2975 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 692962 T2975 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 692963 T2975 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 692966 T2975 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 692967 T2975 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:41905
   [junit4]   2> 692968 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 692969 T2975 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 692969 T2975 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327
   [junit4]   2> 692969 T2975 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/'
   [junit4]   2> 692979 T2975 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solr.xml
   [junit4]   2> 693013 T2975 oasc.CoreContainer.<init> New CoreContainer 503805
   [junit4]   2> 693013 T2975 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/]
   [junit4]   2> 693014 T2975 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 693015 T2975 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 693015 T2975 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 693015 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 693016 T2975 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 693016 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 693016 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 693017 T2975 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 693017 T2975 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 693021 T2975 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 693021 T2975 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 693021 T2975 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 693022 T2975 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35350/solr
   [junit4]   2> 693022 T2975 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 693023 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 693024 T3019 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a3d6ea name:ZooKeeperConnection Watcher:127.0.0.1:35350 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 693024 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 693026 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 693027 T3021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@aaaeb4 name:ZooKeeperConnection Watcher:127.0.0.1:35350/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 693027 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 693031 T2975 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 694033 T2975 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41905_
   [junit4]   2> 694036 T2975 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41905_
   [junit4]   2> 694041 T3008 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 694041 T3021 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 694041 T3003 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 694048 T2975 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 694048 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 694253 T2975 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 694254 T2975 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 694256 T2975 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 694257 T2975 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:38784
   [junit4]   2> 694259 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 694259 T2975 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 694260 T2975 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617
   [junit4]   2> 694260 T2975 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/'
   [junit4]   2> 694275 T2975 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solr.xml
   [junit4]   2> 694345 T2975 oasc.CoreContainer.<init> New CoreContainer 3299526
   [junit4]   2> 694345 T2975 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/]
   [junit4]   2> 694347 T2975 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 694347 T2975 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 694348 T2975 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 694348 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 694349 T2975 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 694349 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 694350 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 694350 T2975 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 694351 T2975 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 694356 T2975 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 694357 T2975 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 694357 T2975 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 694358 T2975 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35350/solr
   [junit4]   2> 694358 T2975 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 694359 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694361 T3033 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aadcf2 name:ZooKeeperConnection Watcher:127.0.0.1:35350 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694361 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694363 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694363 T3035 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16aa9a6 name:ZooKeeperConnection Watcher:127.0.0.1:35350/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694364 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694369 T2975 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 695374 T2975 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38784_
   [junit4]   2> 695376 T2975 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38784_
   [junit4]   2> 695382 T3008 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 695382 T3021 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 695382 T3035 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 695382 T3003 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 695399 T2975 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 695400 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 695605 T2975 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 695606 T2975 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 695609 T2975 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 695610 T2975 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:52590
   [junit4]   2> 695612 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 695612 T2975 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 695612 T2975 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393281958969
   [junit4]   2> 695613 T2975 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393281958969/'
   [junit4]   2> 695622 T2975 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393281958969/solr.xml
   [junit4]   2> 695658 T2975 oasc.CoreContainer.<init> New CoreContainer 6506268
   [junit4]   2> 695659 T2975 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393281958969/]
   [junit4]   2> 695660 T2975 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 695660 T2975 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 695660 T2975 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 695661 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 695661 T2975 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 695661 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 695661 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 695662 T2975 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 695662 T2975 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 695666 T2975 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 695666 T2975 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 695666 T2975 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 695667 T2975 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35350/solr
   [junit4]   2> 695667 T2975 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 695668 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695670 T3047 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13054c5 name:ZooKeeperConnection Watcher:127.0.0.1:35350 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695670 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695672 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695673 T3049 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1afe479 name:ZooKeeperConnection Watcher:127.0.0.1:35350/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695673 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695676 T2975 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696679 T2975 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52590_
   [junit4]   2> 696681 T2975 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52590_
   [junit4]   2> 696686 T3008 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 696687 T3035 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 696687 T3021 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 696687 T3003 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 696687 T3049 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 696694 T2975 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 696695 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 696910 T2975 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 696911 T2975 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 696914 T2975 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 696915 T2975 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58093
   [junit4]   2> 696916 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 696917 T2975 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 696917 T2975 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393281960266
   [junit4]   2> 696917 T2975 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393281960266/'
   [junit4]   2> 696927 T2975 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393281960266/solr.xml
   [junit4]   2> 696963 T2975 oasc.CoreContainer.<init> New CoreContainer 13782384
   [junit4]   2> 696963 T2975 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393281960266/]
   [junit4]   2> 696964 T2975 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 696965 T2975 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 696965 T2975 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 696965 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 696966 T2975 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 696966 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 696966 T2975 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 696967 T2975 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 696967 T2975 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 696971 T2975 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 696971 T2975 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 696972 T2975 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 696972 T2975 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35350/solr
   [junit4]   2> 696972 T2975 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 696973 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696995 T3061 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17f99aa name:ZooKeeperConnection Watcher:127.0.0.1:35350 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696995 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696998 T2975 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696999 T3063 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e80da8 name:ZooKeeperConnection Watcher:127.0.0.1:35350/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696999 T2975 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 697007 T2975 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 698010 T2975 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58093_
   [junit4]   2> 698012 T2975 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58093_
   [junit4]   2> 698015 T3021 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 698015 T3008 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 698015 T3049 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 698015 T3003 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 698015 T3063 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 698015 T3035 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 698021 T2975 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 698022 T2975 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 698104 T3017 oasha.CollectionsHandler.handleCreateAction Creating Collection : action=CREATE&wt=javabin&replicationFactor=2&numShards=2&version=2&collection.configName=conf1&router.name=compositeId&router.field=myOwnField&name=solrj_collection
   [junit4]   2> 698106 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698107 T3005 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.name":"compositeId",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 698107 T3005 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.name":"compositeId",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 698108 T3005 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 698108 T3005 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 698111 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698113 T3004 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 698113 T3004 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 698113 T3004 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 698116 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698117 T3003 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698118 T3049 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698118 T3063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698118 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698117 T3035 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698117 T3008 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698212 T3005 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 698212 T3005 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:38784_
   [junit4]   2> 698214 T3005 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:41905_
   [junit4]   2> 698217 T3005 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:52590_
   [junit4]   2> 698219 T3005 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:58093_
   [junit4]   2> 698302 T3030 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 698302 T3015 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 698302 T3043 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 698303 T3030 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 698303 T3015 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 698303 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698309 T3043 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 698310 T3056 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 698310 T3004 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 698311 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698311 T3056 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 698311 T3004 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38784",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38784_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 698314 T3004 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:41905",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41905_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 698316 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698317 T3004 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:52590",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52590_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 698319 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698320 T3004 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58093",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58093_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 698322 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 698425 T3049 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698425 T3003 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698425 T3063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698425 T3035 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698425 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 698425 T3008 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 699303 T3030 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 699304 T3015 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 699304 T3030 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solr.xml
   [junit4]   2> 699304 T3015 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solr.xml
   [junit4]   2> 699305 T3030 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solrj_collection_shard1_replica1
   [junit4]   2> 699305 T3030 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 699305 T3015 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solrj_collection_shard1_replica2
   [junit4]   2> 699306 T3015 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 699306 T3030 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 699306 T3030 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 699306 T3015 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 699307 T3015 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 699307 T3030 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 699308 T3030 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solrj_collection_shard1_replica1/'
   [junit4]   2> 699308 T3015 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 699308 T3015 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solrj_collection_shard1_replica2/'
   [junit4]   2> 699310 T3043 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 699311 T3043 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393281958969/solr.xml
   [junit4]   2> 699311 T3043 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393281958969/solrj_collection_shard2_replica1
   [junit4]   2> 699311 T3043 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 699311 T3056 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 699312 T3043 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 699313 T3043 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 699312 T3056 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393281960266/solr.xml
   [junit4]   2> 699313 T3056 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393281960266/solrj_collection_shard2_replica2
   [junit4]   2> 699313 T3056 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 699313 T3043 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 699314 T3043 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393281958969/solrj_collection_shard2_replica1/'
   [junit4]   2> 699318 T3056 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 699318 T3056 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 699319 T3056 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 699320 T3056 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393281960266/solrj_collection_shard2_replica2/'
   [junit4]   2> 699342 T3030 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 699351 T3015 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 699358 T3043 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 699373 T3056 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 699383 T3030 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 699385 T3030 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 699392 T3030 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 699396 T3043 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 699397 T3043 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 699404 T3043 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 699408 T3015 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 699410 T3015 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 699446 T3015 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 699461 T3056 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 699464 T3056 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 699475 T3056 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 699802 T3030 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 699807 T3030 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 699810 T3030 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 699821 T3030 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 699824 T3030 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 699828 T3030 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 699828 T3015 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 699829 T3030 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 699829 T3030 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 699829 T3030 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 699830 T3030 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 699831 T3030 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 699831 T3030 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 699831 T3030 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solrj_collection_shard1_replica1/data/
   [junit4]   2> 699832 T3030 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1114305
   [junit4]   2> 699832 T3030 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solrj_collection_shard1_replica1/data
   [junit4]   2> 699833 T3030 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 699833 T3030 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 699833 T3015 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 699833 T3030 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 699834 T3030 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6183218002126731]
   [junit4]   2> 699835 T3030 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@12d1354 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@43c124),segFN=segments_1,generation=1}
   [junit4]   2> 699835 T3030 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 699836 T3015 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 699838 T3030 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 699839 T3030 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 699839 T3030 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 699840 T3030 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 699840 T3030 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 699841 T3030 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 699841 T3030 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 699842 T3030 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 699842 T3030 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 699842 T3030 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 699843 T3030 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 699843 T3030 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 699844 T3030 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 699844 T3030 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 699845 T3030 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 699845 T3030 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 699845 T3015 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 699846 T3030 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 699848 T3030 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 699849 T3015 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 699851 T3030 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 699851 T3030 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 699852 T3015 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 699852 T3030 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.359375, noCFSRatio=0.3485513564594369]
   [junit4]   2> 699853 T3015 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 699853 T3030 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@12d1354 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@43c124),segFN=segments_1,generation=1}
   [junit4]   2> 699853 T3015 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 699853 T3030 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 699854 T3015 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 699854 T3030 oass.SolrIndexSearcher.<init> Opening Searcher@8f9d7d[solrj_collection_shard1_replica1] main
   [junit4]   2> 699855 T3015 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 699855 T3015 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 699856 T3015 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 699856 T3015 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solrj_collection_shard1_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solrj_collection_shard1_replica2/data/
   [junit4]   2> 699856 T3015 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1114305
   [junit4]   2> 699857 T3069 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica1] Registered new searcher Searcher@8f9d7d[solrj_collection_shard1_replica1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 699857 T3015 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solrj_collection_shard1_replica2/data
   [junit4]   2> 699857 T3015 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 699858 T3015 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 699858 T3015 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 699858 T3015 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6183218002126731]
   [junit4]   2> 699859 T3030 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica1
   [junit4]   2> 699859 T3030 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica1 address:https://127.0.0.1:38784 collection:solrj_collection shard:shard1
   [junit4]   2> 699859 T3015 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1ddf6e2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1af340a),segFN=segments_1,generation=1}
   [junit4]   2> 699860 T3015 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 699860 T3030 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 699863 T3015 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 699864 T3015 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 699864 T3015 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 699864 T3015 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 699865 T3015 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 699865 T3015 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 699865 T3015 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 699866 T3015 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 699866 T3015 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 699866 T3030 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 699866 T3015 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 699867 T3015 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 699867 T3015 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 699868 T3015 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 699868 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 699868 T3015 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 699868 T3030 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=0
   [junit4]   2> 699869 T3030 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
   [junit4]   2> 699869 T3030 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 699868 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C1514 name=solrj_collection_shard1_replica1 org.apache.solr.core.SolrCore@3c8ad3 url=https://127.0.0.1:38784/solrj_collection_shard1_replica1 node=127.0.0.1:38784_ C1514_STATE=coll:solrj_collection core:solrj_collection_shard1_replica1 props:{state=down, base_url=https://127.0.0.1:38784, core=solrj_collection_shard1_replica1, node_name=127.0.0.1:38784_}
   [junit4]   2> 699870 T3030 C1514 P38784 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:38784/solrj_collection_shard1_replica1/
   [junit4]   2> 699869 T3004 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 699869 T3015 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 699870 T3030 C1514 P38784 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=https://127.0.0.1:38784 START replicas=[https://127.0.0.1:41905/solrj_collection_shard1_replica2/] nUpdates=100
   [junit4]   2> 699870 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 699871 T3030 C1514 P38784 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica1 url=https://127.0.0.1:38784 DONE.  We have no versions.  sync failed.
   [junit4]   2> 699872 T3030 C1514 P38784 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 699871 T3015 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 699873 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 699872 T3030 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 699873 T3015 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 699873 T3030 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:38784/solrj_collection_shard1_replica1/ shard1
   [junit4]   2> 699874 T3030 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard1
   [junit4]   2> 699875 T3015 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 699878 T3015 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 699879 T3015 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 699879 T3015 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.359375, noCFSRatio=0.3485513564594369]
   [junit4]   2> 699880 T3015 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1ddf6e2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1af340a),segFN=segments_1,generation=1}
   [junit4]   2> 699880 T3015 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 699881 T3015 oass.SolrIndexSearcher.<init> Opening Searcher@1151763[solrj_collection_shard1_replica2] main
   [junit4]   2> 699881 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 699884 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 699884 T3072 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@1151763[solrj_collection_shard1_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 699887 T3015 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 699887 T3015 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:https://127.0.0.1:41905 collection:solrj_collection shard:shard1
   [junit4]   2> ASYNC  NEW_CORE C1515 name=solrj_collection_shard1_replica2 org.apache.solr.core.SolrCore@68826d url=https://127.0.0.1:41905/solrj_collection_shard1_replica2 node=127.0.0.1:41905_ C1515_STATE=coll:solrj_collection core:solrj_collection_shard1_replica2 props:{state=down, base_url=https://127.0.0.1:41905, core=solrj_collection_shard1_replica2, node_name=127.0.0.1:41905_}
   [junit4]   2> 699967 T3014 C1515 P41905 oasc.SolrCore.execute [solrj_collection_shard1_replica2] webapp= path=/get params={wt=javabin&getVersions=100&version=2&distrib=false&qt=/get} status=0 QTime=1 
   [junit4]   2> 699986 T3049 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 699986 T3035 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 699986 T3063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 699986 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 699986 T3003 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 699986 T3008 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 699991 T3043 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 699998 T3043 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 700002 T3043 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 700005 T3015 oasc.ZkController.register We are https://127.0.0.1:41905/solrj_collection_shard1_replica2/ and leader is https://127.0.0.1:38784/solrj_collection_shard1_replica1/
   [junit4]   2> 700005 T3015 oasc.ZkController.register No LogReplay needed for core=solrj_collection_shard1_replica2 baseURL=https://127.0.0.1:41905
   [junit4]   2> 700006 T3015 oasc.ZkController.checkRecovery Core needs to recover:solrj_collection_shard1_replica2
   [junit4]   2> 700006 T3015 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> 700007 T3074 C1515 P41905 oasc.RecoveryStrategy.run Starting recovery process.  core=solrj_collection_shard1_replica2 recoveringAfterStartup=true
   [junit4]   2> 700007 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 700007 T3074 C1515 P41905 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 700008 T3074 C1515 P41905 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=recovering
   [junit4]   2> 700009 T3015 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393281956327/solr.xml
   [junit4]   2> 700009 T3015 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=CREATE&wt=javabin&numShards=2&shard=shard1&version=2&collection=solrj_collection&collection.configName=conf1&qt=/admin/cores&name=solrj_collection_shard1_replica2} status=0 QTime=1708 
   [junit4]   2> 700032 T3030 oasc.ZkController.register We are https://127.0.0.1:38784/solrj_collection_shard1_replica1/ and leader is https://127.0.0.1:38784/solrj_collection_shard1_replica1/
   [junit4]   2> 700033 T3030 oasc.ZkController.register No LogReplay needed for core=solrj_collection_shard1_replica1 baseURL=https://127.0.0.1:38784
   [junit4]   2> 700033 T3030 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 700033 T3030 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=active
   [junit4]   2> 700033 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 700034 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 700034 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 700036 T3056 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 700037 T3004 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 700037 T3030 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 700038 T3004 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:41905",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41905_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 700040 T3030 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393281957617/solr.xml
   [junit4]   2> 700040 T3030 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=CREATE&wt=javabin&numShards=2&shard=shard1&version=2&collection=solrj_collection&collection.configName=conf1&qt=/admin/cores&name=solrj_collection_shard1_replica1} status=0 QTime=1740 
   [junit4]   2> 700042 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 700042 T3004 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38784",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38784_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 700043 T3056 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 700046 T3056 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 700048 T3003 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 700050 T3043 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 700054 T3043 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 700058 T3043 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 700059 T3043 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 700060 T3043 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 700060 T3043 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 700060 T3056 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 700061 T3043 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 700061 T3043 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 700062 T3043 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 700062 T3043 oasc.SolrCore.<init> [solrj_collection_shard2_replica

[...truncated too long message...]

   [junit4]   2> 	  66) Thread[id=614, name=commitScheduler-393-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  67) Thread[id=425, name=coreZkRegister-330-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  68) Thread[id=725, name=HashSessionScavenger-23, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 	  69) Thread[id=847, name=qtp16362332-847 Acceptor0 SelectChannelConnector@127.0.0.1:38670, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  70) Thread[id=742, name=searcherExecutor-450-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  71) Thread[id=449, name=qtp13994440-449 Acceptor1 SelectChannelConnector@127.0.0.1:39111, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  72) Thread[id=547, name=qtp4268729-547 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  73) Thread[id=559, name=searcherExecutor-390-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  74) Thread[id=445, name=HashSessionScavenger-11, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=iw, timezone=Europe/Monaco
   [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=86,free=148675952,total=344981504
   [junit4]   2> NOTE: All tests run in this JVM: [SpellCheckComponentTest, TestFieldTypeCollectionResource, HighlighterTest, TestRecovery, TestComponentsName, OutputWriterTest, TestSolrDeletionPolicy2, TestPartialUpdateDeduplication, IndexBasedSpellCheckerTest, OpenExchangeRatesOrgProviderTest, UniqFieldsUpdateProcessorFactoryTest, TestFastLRUCache, DeleteReplicaTest, TestUtils, TestJmxIntegration, JSONWriterTest, TestZkChroot, XsltUpdateRequestHandlerTest, HdfsBasicDistributedZkTest, HdfsUnloadDistributedZkTest, SolrTestCaseJ4Test, TestNumberUtils, TestQuerySenderNoQuery, SuggestComponentTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=5135ECA5636D93E7 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw -Dtests.timezone=Europe/Monaco -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5135ECA5636D93E7]:0)
   [junit4] Completed on J1 in 7224.90s, 1 test, 2 errors <<< FAILURES!

[...truncated 213 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1606 tests, 1 suite-level error, 1 error, 1 failure, 250 ignored (1 assumption)

Total time: 147 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b04) - Build # 9585 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9585/
Java: 32bit/jdk1.7.0_60-ea-b04 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Could not find collection:addReplicaColl

Stack Trace:
java.lang.AssertionError: Could not find collection:addReplicaColl
	at __randomizedtesting.SeedInfo.seed([C3351F815D038A65:42D391992A5CEA59]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:147)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:724)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1193)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:213)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10571 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 703189 T3529 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 703189 T3529 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ps_e/dr
   [junit4]   2> 703195 T3529 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1393269117204
   [junit4]   2> 703196 T3529 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 703196 T3530 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 703296 T3529 oasc.ZkTestServer.run start zk server on port:38886
   [junit4]   2> 703298 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 703954 T3536 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b5a14 name:ZooKeeperConnection Watcher:127.0.0.1:38886 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 703955 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 703956 T3529 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 703963 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 703965 T3538 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e5e6e3 name:ZooKeeperConnection Watcher:127.0.0.1:38886/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 703965 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 703966 T3529 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 703970 T3529 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 703975 T3529 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 703978 T3529 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 703981 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 703982 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 703988 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 703989 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 703994 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 703995 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 703999 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 703999 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 704004 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 704004 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 704008 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 704009 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 704013 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 704014 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 704018 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 704019 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 704022 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 704023 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 704028 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 704029 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 704034 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 704035 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 704041 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 704043 T3540 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18d7c50 name:ZooKeeperConnection Watcher:127.0.0.1:38886/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 704043 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 704047 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 704049 T3542 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d99ec6 name:ZooKeeperConnection Watcher:127.0.0.1:38886 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 704049 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 704050 T3529 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 704055 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 704056 T3544 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1131027 name:ZooKeeperConnection Watcher:127.0.0.1:38886/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 704057 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 704058 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 704059 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 704065 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 704066 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 704072 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 704072 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 704077 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 704078 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 704082 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 704083 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 704087 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 704088 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 704094 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 704094 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 704099 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 704100 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 704105 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 704105 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 704110 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 704110 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 704115 T3529 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 704115 T3529 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 704327 T3529 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 704329 T3529 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 704332 T3529 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36398
   [junit4]   2> 704333 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 704334 T3529 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 704334 T3529 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393269118129
   [junit4]   2> 704335 T3529 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393269118129/'
   [junit4]   2> 704351 T3529 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393269118129/solr.xml
   [junit4]   2> 704371 T3529 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393269118129
   [junit4]   2> 704372 T3529 oasc.CoreContainer.<init> New CoreContainer 14740147
   [junit4]   2> 704372 T3529 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393269118129/]
   [junit4]   2> 704373 T3529 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 704373 T3529 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 704373 T3529 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 704374 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 704374 T3529 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 704374 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 704374 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 704375 T3529 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 704375 T3529 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 704378 T3529 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 704378 T3529 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 704379 T3529 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 704379 T3529 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38886/solr
   [junit4]   2> 704379 T3529 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 704380 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 704381 T3555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10f962d name:ZooKeeperConnection Watcher:127.0.0.1:38886 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 704382 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 704385 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 704387 T3557 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1755ae8 name:ZooKeeperConnection Watcher:127.0.0.1:38886/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 704387 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 704389 T3529 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 704393 T3529 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 704396 T3529 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 704397 T3529 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36398_ps_e%2Fdr
   [junit4]   2> 704398 T3529 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36398_ps_e%2Fdr
   [junit4]   2> 704400 T3529 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 704404 T3529 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 704411 T3529 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36398_ps_e%2Fdr
   [junit4]   2> 704412 T3529 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 704414 T3529 oasc.Overseer.start Overseer (id=91309284865277958-127.0.0.1:36398_ps_e%2Fdr-n_0000000000) starting
   [junit4]   2> 704418 T3529 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 704424 T3559 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 704425 T3529 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 704426 T3559 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 704429 T3529 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 704432 T3529 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 704437 T3558 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 704439 T3529 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393269118129
   [junit4]   2> 704448 T3529 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1393269118129/conf/
   [junit4]   2> 704529 T3529 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 704530 T3529 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 704530 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 704532 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 704534 T3562 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d3ed37 name:ZooKeeperConnection Watcher:127.0.0.1:38886/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 704534 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 704536 T3529 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 704539 T3529 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 704743 T3529 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 704743 T3529 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 704746 T3529 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 704747 T3529 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42908
   [junit4]   2> 704749 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 704750 T3529 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 704751 T3529 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547
   [junit4]   2> 704751 T3529 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/'
   [junit4]   2> 704766 T3529 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/solr.xml
   [junit4]   2> 704786 T3529 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547
   [junit4]   2> 704786 T3529 oasc.CoreContainer.<init> New CoreContainer 19406913
   [junit4]   2> 704787 T3529 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/]
   [junit4]   2> 704787 T3529 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 704788 T3529 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 704788 T3529 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 704788 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 704788 T3529 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 704789 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 704789 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 704789 T3529 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 704789 T3529 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 704792 T3529 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 704793 T3529 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 704793 T3529 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 704793 T3529 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38886/solr
   [junit4]   2> 704793 T3529 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 704794 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 704795 T3573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18e6307 name:ZooKeeperConnection Watcher:127.0.0.1:38886 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 704796 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 704799 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 704800 T3575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cfbe65 name:ZooKeeperConnection Watcher:127.0.0.1:38886/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 704801 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 704807 T3529 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 705811 T3529 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42908_ps_e%2Fdr
   [junit4]   2> 705813 T3529 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42908_ps_e%2Fdr
   [junit4]   2> 705818 T3557 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 705818 T3575 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 705818 T3562 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 705828 T3529 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547
   [junit4]   2> 705840 T3529 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/conf/
   [junit4]   2> 705933 T3529 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 705934 T3529 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 705935 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 706129 T3529 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 706130 T3529 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 706133 T3529 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 706134 T3529 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:46395
   [junit4]   2> 706136 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 706137 T3529 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 706137 T3529 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945
   [junit4]   2> 706138 T3529 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/'
   [junit4]   2> 706152 T3529 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/solr.xml
   [junit4]   2> 706172 T3529 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945
   [junit4]   2> 706172 T3529 oasc.CoreContainer.<init> New CoreContainer 33053259
   [junit4]   2> 706173 T3529 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/]
   [junit4]   2> 706173 T3529 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 706174 T3529 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 706174 T3529 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 706174 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 706175 T3529 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 706175 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 706175 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 706175 T3529 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 706176 T3529 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 706179 T3529 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 706179 T3529 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 706179 T3529 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 706180 T3529 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38886/solr
   [junit4]   2> 706180 T3529 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 706181 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 706182 T3587 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9761e3 name:ZooKeeperConnection Watcher:127.0.0.1:38886 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706183 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 706187 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 706188 T3589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1602fb0 name:ZooKeeperConnection Watcher:127.0.0.1:38886/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706189 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 706196 T3529 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 707200 T3529 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46395_ps_e%2Fdr
   [junit4]   2> 707202 T3529 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46395_ps_e%2Fdr
   [junit4]   2> 707204 T3589 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 707205 T3557 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 707204 T3562 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 707204 T3575 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 707224 T3529 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945
   [junit4]   2> 707230 T3529 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/conf/
   [junit4]   2> 707324 T3529 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 707325 T3529 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 707325 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 707521 T3529 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 707522 T3529 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 707525 T3529 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 707526 T3529 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:55838
   [junit4]   2> 707528 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 707528 T3529 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 707529 T3529 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334
   [junit4]   2> 707530 T3529 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/'
   [junit4]   2> 707543 T3529 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/solr.xml
   [junit4]   2> 707564 T3529 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334
   [junit4]   2> 707565 T3529 oasc.CoreContainer.<init> New CoreContainer 23462482
   [junit4]   2> 707565 T3529 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/]
   [junit4]   2> 707565 T3529 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 707566 T3529 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 707566 T3529 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 707566 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 707567 T3529 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 707567 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 707567 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 707567 T3529 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 707568 T3529 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 707571 T3529 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 707571 T3529 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 707571 T3529 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 707572 T3529 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38886/solr
   [junit4]   2> 707572 T3529 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 707573 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 707575 T3601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15bf7ff name:ZooKeeperConnection Watcher:127.0.0.1:38886 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 707576 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 707579 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 707580 T3603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19c57fb name:ZooKeeperConnection Watcher:127.0.0.1:38886/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 707580 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 707590 T3529 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 708595 T3529 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55838_ps_e%2Fdr
   [junit4]   2> 708597 T3529 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55838_ps_e%2Fdr
   [junit4]   2> 708602 T3562 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 708602 T3589 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 708603 T3575 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 708602 T3557 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 708603 T3603 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 708610 T3529 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334
   [junit4]   2> 708622 T3529 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/conf/
   [junit4]   2> 708714 T3529 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 708715 T3529 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 708716 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 708928 T3529 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 708929 T3529 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 708932 T3529 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 708933 T3529 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:43524
   [junit4]   2> 708935 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 708936 T3529 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 708937 T3529 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726
   [junit4]   2> 708937 T3529 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/'
   [junit4]   2> 708950 T3529 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/solr.xml
   [junit4]   2> 708969 T3529 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726
   [junit4]   2> 708969 T3529 oasc.CoreContainer.<init> New CoreContainer 27213600
   [junit4]   2> 708970 T3529 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/]
   [junit4]   2> 708970 T3529 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 708971 T3529 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 708971 T3529 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 708971 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 708971 T3529 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 708972 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 708972 T3529 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 708972 T3529 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 708973 T3529 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 708975 T3529 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 708976 T3529 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 708976 T3529 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 708976 T3529 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38886/solr
   [junit4]   2> 708977 T3529 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 708977 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 708979 T3615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10b52d8 name:ZooKeeperConnection Watcher:127.0.0.1:38886 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 708980 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 708982 T3529 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 708984 T3617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e52436 name:ZooKeeperConnection Watcher:127.0.0.1:38886/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 708984 T3529 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 708988 T3529 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 709992 T3529 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43524_ps_e%2Fdr
   [junit4]   2> 709994 T3529 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43524_ps_e%2Fdr
   [junit4]   2> 710000 T3603 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 710000 T3562 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 710000 T3575 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 710001 T3557 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 710000 T3617 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 710000 T3589 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 710007 T3529 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726
   [junit4]   2> 710019 T3529 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/conf/
   [junit4]   2> 710115 T3529 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 710115 T3529 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 710116 T3529 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 710209 T3569 oasha.CollectionsHandler.handleCreateAction Creating Collection : action=CREATE&version=2&name=solrj_collection&collection.configName=conf1&router.field=myOwnField&wt=javabin&router.name=compositeId&numShards=2&replicationFactor=2
   [junit4]   2> 710212 T3557 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 710214 T3559 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 710214 T3559 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 710216 T3559 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 710216 T3559 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 710222 T3557 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 710224 T3558 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 710226 T3558 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 710226 T3558 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 710229 T3557 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 710231 T3575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710231 T3557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710231 T3603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710231 T3562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710231 T3589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710231 T3617 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710322 T3559 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 710323 T3559 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:55838_ps_e%2Fdr
   [junit4]   2> 710326 T3559 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:42908_ps_e%2Fdr
   [junit4]   2> 710331 T3559 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:46395_ps_e%2Fdr
   [junit4]   2> 710336 T3559 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:43524_ps_e%2Fdr
   [junit4]   2> 710417 T3584 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 710418 T3557 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 710418 T3584 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 710419 T3558 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 710419 T3558 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46395/ps_e/dr",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46395_ps_e%2Fdr",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 710426 T3557 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 710426 T3613 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 710427 T3613 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 710427 T3557 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 710428 T3597 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 710428 T3558 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43524/ps_e/dr",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43524_ps_e%2Fdr",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 710429 T3557 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 710429 T3597 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 710430 T3571 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 710430 T3571 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 710431 T3558 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55838/ps_e/dr",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55838_ps_e%2Fdr",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 710435 T3557 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 710436 T3558 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42908/ps_e/dr",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42908_ps_e%2Fdr",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 710438 T3557 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 710541 T3589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710541 T3617 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710541 T3557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710541 T3562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710541 T3575 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 710541 T3603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 711418 T3584 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 711420 T3584 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/solrj_collection_shard2_replica1
   [junit4]   2> 711420 T3584 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 711421 T3584 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 711422 T3584 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 711424 T3584 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 711424 T3584 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/solrj_collection_shard2_replica1/'
   [junit4]   2> 711428 T3613 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 711429 T3613 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/solrj_collection_shard2_replica2
   [junit4]   2> 711430 T3613 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 711429 T3597 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 711431 T3571 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 711431 T3613 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 711432 T3613 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 711432 T3597 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/solrj_collection_shard1_replica1
   [junit4]   2> 711432 T3571 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/solrj_collection_shard1_replica2
   [junit4]   2> 711433 T3597 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 711434 T3613 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 711434 T3571 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 711435 T3597 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 711436 T3597 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 711435 T3613 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/solrj_collection_shard2_replica2/'
   [junit4]   2> 711436 T3571 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 711437 T3571 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 711439 T3597 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 711439 T3571 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 711440 T3597 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/solrj_collection_shard1_replica1/'
   [junit4]   2> 711440 T3571 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/solrj_collection_shard1_replica2/'
   [junit4]   2> 711485 T3584 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 711494 T3613 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 711496 T3571 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 711498 T3597 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 711520 T3584 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 711521 T3584 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 711527 T3613 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 711528 T3584 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 711529 T3613 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 711531 T3571 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 711532 T3571 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 711533 T3597 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 711534 T3597 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 711535 T3613 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 711538 T3571 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 711540 T3597 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 711795 T3613 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 711796 T3571 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 711800 T3613 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 711800 T3571 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 711802 T3613 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 711802 T3571 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 711820 T3613 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711820 T3571 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711823 T3613 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711823 T3571 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711827 T3613 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711827 T3571 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711828 T3613 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711828 T3571 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711831 T3571 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711831 T3584 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 711830 T3613 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711832 T3571 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711832 T3613 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711833 T3571 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711833 T3613 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711834 T3613 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711833 T3571 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711834 T3613 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 711834 T3571 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 711835 T3613 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/solrj_collection_shard2_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/solrj_collection_shard2_replica2/data/
   [junit4]   2> 711835 T3571 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/solrj_collection_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/solrj_collection_shard1_replica2/data/
   [junit4]   2> 711835 T3613 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1168a4c
   [junit4]   2> 711835 T3571 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1168a4c
   [junit4]   2> 711836 T3584 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 711836 T3571 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/solrj_collection_shard1_replica2/data
   [junit4]   2> 711837 T3613 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/solrj_collection_shard2_replica2/data
   [junit4]   2> 711837 T3571 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 711837 T3613 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 711837 T3571 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 711838 T3584 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 711838 T3613 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 711838 T3571 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1393269118547/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 711839 T3613 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1393269122726/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 711839 T3571 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.29156805261700114]
   [junit4]   2> 711840 T3613 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.29156805261700114]
   [junit4]   2> 711841 T3597 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 711840 T3571 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@17c7488 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@190e77a),segFN=segments_1,generation=1}
   [junit4]   2> 711841 T3613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@566ec4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a94b3b),segFN=segments_1,generation=1}
   [junit4]   2> 711841 T3613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 711841 T3571 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 711845 T3597 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 711846 T3613 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 711846 T3613 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 711846 T3571 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 711846 T3613 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 711847 T3571 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 711847 T3597 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 711847 T3571 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 711847 T3613 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 711848 T3571 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 711848 T3613 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 711849 T3571 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 711849 T3613 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 711849 T3571 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 711850 T3613 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 711850 T3571 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 711850 T3613 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 711851 T3571 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 711851 T3613 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 711851 T3571 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 711851 T3613 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 711852 T3571 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 711852 T3613 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 711852 T3613 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 711853 T3584 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711852 T3571 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 711853 T3613 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 711853 T3571 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 711854 T3613 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 711854 T3571 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 711855 T3613 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 711855 T3571 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 711855 T3613 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 711856 T3571 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 711856 T3613 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 711856 T3571 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 711856 T3584 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711857 T3571 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 711858 T3571 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 711858 T3613 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 711858 T3597 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711859 T3584 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711860 T3584 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711861 T3613 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 711861 T3584 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711860 T3571 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 711861 T3584 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711861 T3613 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 711862 T3597 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711862 T3584 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711862 T3571 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 711863 T3584 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711863 T3571 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2632361204656867]
   [junit4]   2> 711863 T3613 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2632361204656867]
   [junit4]   2> 711864 T3571 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@17c7488 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@190e77a),segFN=segments_1,generation=1}
   [junit4]   2> 711864 T3613 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@566ec4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a94b3b),segFN=segments_1,generation=1}
   [junit4]   2> 711863 T3584 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 711865 T3597 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711865 T3584 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/solrj_collection_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/solrj_collection_shard2_replica1/data/
   [junit4]   2> 711865 T3613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 711866 T3597 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711866 T3597 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711864 T3571 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 711866 T3597 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711866 T3613 oass.SolrIndexSearcher.<init> Opening Searcher@1052d[solrj_collection_shard2_replica2] main
   [junit4]   2> 711865 T3584 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1168a4c
   [junit4]   2> 711867 T3571 oass.SolrIndexSearcher.<init> Opening Searcher@bf0036[solrj_collection_shard1_replica2] main
   [junit4]   2> 711868 T3597 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711869 T3597 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711868 T3584 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/solrj_collection_shard2_replica1/data
   [junit4]   2> 711869 T3597 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 711869 T3584 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 711870 T3597 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/solrj_collection_shard1_replica1/data/
   [junit4]   2> 711870 T3623 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@bf0036[solrj_collection_shard1_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 711870 T3624 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@1052d[solrj_collection_shard2_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 711870 T3584 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 711870 T3597 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1168a4c
   [junit4]   2> 711871 T3584 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1393269119945/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 711872 T3571 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 711872 T3584 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.29156805261700114]
   [junit4]   2> 711872 T3597 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/solrj_collection_shard1_replica1/data
   [junit4]   2> 711872 T3597 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1393269121334/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 711872 T3571 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:https://127.0.0.1:42908/ps_e/dr collection:solrj_collection shard:shard1
   [junit4]   2> 711873 T3613 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
   [junit4]   2> 711873 T3597 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/w

[...truncated too long message...]

mon.cloud.SolrZkClient.exists(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:414)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 847164 T3617 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/acollectionafterbaddelete/leader_elect/shard1/election/91309284865277967-core_node2-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 847165 T3617 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 847166 T3617 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 847166 T3617 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 847166 T3617 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 847167 T3617 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/nodes_used_collection/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=de_AT, timezone=America/Chicago
   [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=122706232,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestRecovery, FileUtilsTest, SolrIndexConfigTest, TestQuerySenderListener, CoreContainerCoreInitFailuresTest, TestSweetSpotSimilarityFactory, TestCSVLoader, EnumFieldTest, SignatureUpdateProcessorFactoryTest, TestCoreDiscovery, FieldAnalysisRequestHandlerTest, DeleteInactiveReplicaTest, XsltUpdateRequestHandlerTest, NoCacheHeaderTest, TestSchemaResource, HdfsBasicDistributedZk2Test, OpenCloseCoreStressTest, TestIndexingPerformance, SortByFunctionTest, DeleteReplicaTest, XmlUpdateRequestHandlerTest, SuggesterWFSTTest, TestSolrXmlPersistence, ConvertedLegacyTest, TestOmitPositions, TestXIncludeConfig, TestQuerySenderNoQuery, PluginInfoTest, OpenExchangeRatesOrgProviderTest, TestRecoveryHdfs, SolrTestCaseJ4Test, TestAtomicUpdateErrorCases, PingRequestHandlerTest, OverseerRolesTest, BadIndexSchemaTest, SolrPluginUtilsTest, TestSolr4Spatial, TestUpdate, UniqFieldsUpdateProcessorFactoryTest, SpellCheckCollatorTest, TestDistributedGrouping, HighlighterTest, AlternateDirectoryTest, CustomCollectionTest, TestSchemaSimilarityResource, TestDistributedSearch, TestFuzzyAnalyzedSuggestions, ZkSolrClientTest, TestNRTOpen, HdfsUnloadDistributedZkTest, CurrencyFieldXmlFileTest, FullSolrCloudDistribCmdsTest, TestMaxScoreQueryParser, TestDFRSimilarityFactory, AssignTest, DistributedQueryComponentOptimizationTest, TestNumberUtils, BlockDirectoryTest, OutputWriterTest, TestSimpleQParserPlugin, TestSerializedLuceneMatchVersion, TestCollationField, MinimalSchemaTest, TimeZoneUtilsTest, CachingDirectoryFactoryTest, TestLazyCores, StressHdfsTest, TestModifyConfFiles, TestIBSimilarityFactory, TestWordDelimiterFilterFactory, DateFieldTest, TestFieldResource, MultiTermTest, TestPhraseSuggestions, TestFastWriter, DirectSolrSpellCheckerTest, DirectSolrConnectionTest, TestManagedSchemaFieldResource, TestAnalyzedSuggestions, SuggesterTSTTest, StandardRequestHandlerTest, IndexBasedSpellCheckerTest, DocumentAnalysisRequestHandlerTest, StatsComponentTest, ShardSplitTest, PrimUtilsTest, NotRequiredUniqueKeyTest, JSONWriterTest, TestJmxIntegration, PreAnalyzedUpdateProcessorTest, TestComponentsName, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 144.04s, 1 test, 1 failure <<< FAILURES!

[...truncated 567 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:465: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:445: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 373 suites, 1606 tests, 1 failure, 35 ignored (7 assumptions)

Total time: 49 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b04 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure