You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/28 16:44:53 UTC

[JENKINS] Lucene-Solr-4.x-Linux (32bit/ibm-j9-jdk7) - Build # 9518 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9518/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

3 tests failed.
REGRESSION:  org.apache.solr.handler.component.DistributedQueryElevationComponentTest.testDistribSearch

Error Message:
.responseHeader.params.rows!=forceElevation (unordered or missing)

Stack Trace:
junit.framework.AssertionFailedError: .responseHeader.params.rows!=forceElevation (unordered or missing)
	at __randomizedtesting.SeedInfo.seed([1F78A35E78D2BB56:9E9E2D460F8DDB6A]:0)
	at junit.framework.Assert.fail(Assert.java:50)
	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.DistributedQueryElevationComponentTest.doTest(DistributedQueryElevationComponentTest.java:89)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.handler.component.TermVectorComponentDistributedTest.testDistribSearch

Error Message:
.termVectors.0.test_offtv!=test_basictv (unordered or missing)

Stack Trace:
junit.framework.AssertionFailedError: .termVectors.0.test_offtv!=test_basictv (unordered or missing)
	at __randomizedtesting.SeedInfo.seed([1F78A35E78D2BB56:9E9E2D460F8DDB6A]:0)
	at junit.framework.Assert.fail(Assert.java:50)
	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.TermVectorComponentDistributedTest.doTest(TermVectorComponentDistributedTest.java:133)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection

Error Message:
expected:<1> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<3>
	at __randomizedtesting.SeedInfo.seed([1F78A35E78D2BB56:444858967B95182A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:78)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)




Build Log:
[...truncated 10413 lines...]
   [junit4] Suite: org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest
   [junit4]   2> 248789 T1093 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 248790 T1093 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178
   [junit4]   2> 248791 T1093 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 248793 T1093 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 248794 T1093 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 248795 T1093 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 248834 T1093 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 248878 T1093 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 248879 T1093 oass.IndexSchema.readSchema Reading Solr Schema from schema12.xml
   [junit4]   2> 248884 T1093 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 249184 T1093 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 249186 T1093 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 249188 T1093 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 249195 T1093 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 249198 T1093 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 249201 T1093 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 249202 T1093 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr
   [junit4]   2> 249202 T1093 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 249239 T1093 oasc.CoreContainer.<init> New CoreContainer 613067616
   [junit4]   2> 249239 T1093 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 249240 T1093 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 249240 T1093 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 249241 T1093 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 249241 T1093 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 249242 T1093 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 249242 T1093 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 249242 T1093 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 249243 T1093 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 249243 T1093 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 249248 T1093 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 249248 T1093 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 249249 T1093 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 249252 T1094 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 249252 T1094 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 249254 T1094 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 249254 T1094 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 249255 T1094 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 249301 T1094 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 249345 T1094 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 249345 T1094 oass.IndexSchema.readSchema Reading Solr Schema from schema12.xml
   [junit4]   2> 249351 T1094 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 249635 T1094 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 249637 T1094 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 249639 T1094 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 249646 T1094 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 249649 T1094 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 249653 T1094 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 249653 T1094 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178/
   [junit4]   2> 249653 T1094 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41b14733
   [junit4]   2> 249654 T1094 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178
   [junit4]   2> 249655 T1094 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178/index/
   [junit4]   2> 249655 T1094 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178/index' doesn't exist. Creating new index...
   [junit4]   2> 249655 T1094 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178/index
   [junit4]   2> 249656 T1094 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47138659193377863]
   [junit4]   2> 249657 T1094 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_1,generation=1}
   [junit4]   2> 249657 T1094 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 249657 T1094 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 249659 T1094 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 249660 T1094 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 249660 T1094 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 249660 T1094 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 249661 T1094 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 249661 T1094 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 249661 T1094 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 249662 T1094 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 249662 T1094 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 249662 T1094 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 249663 T1094 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 249663 T1094 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 249664 T1094 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 249664 T1094 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 249664 T1094 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 249665 T1094 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 249665 T1094 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 249665 T1094 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 249666 T1094 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 249666 T1094 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 249667 T1094 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 249667 T1094 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 249667 T1094 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 249668 T1094 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 249668 T1094 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 249669 T1094 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 249669 T1094 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 249669 T1094 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 249670 T1094 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 249670 T1094 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 249671 T1094 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 249671 T1094 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 249671 T1094 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 249672 T1094 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 249672 T1094 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 249673 T1094 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 249674 T1094 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 249675 T1094 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 249677 T1094 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 249677 T1094 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 249677 T1094 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@c0c8827c
   [junit4]   2> 249678 T1094 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_1,generation=1}
   [junit4]   2> 249678 T1094 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 249679 T1094 oass.SolrIndexSearcher.<init> Opening Searcher@340ed953[collection1] main
   [junit4]   2> 249679 T1094 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 249720 T1094 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 249754 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@340ed953[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 249758 T1094 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 249759 T1093 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 249763 T1093 oas.SolrTestCaseJ4.setUp ###Starting testMultiThreaded
   [junit4]   2> ASYNC  NEW_CORE C513 name=collection1 org.apache.solr.core.SolrCore@13c78f74
   [junit4]   2> 249763 T1093 C513 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 249764 T1093 C513 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 249771 T1093 C513 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 249773 T1093 C513 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_2,generation=2}
   [junit4]   2> 249774 T1093 C513 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 249776 T1093 C513 oass.SolrIndexSearcher.<init> Opening Searcher@930fbbda[collection1] main
   [junit4]   2> 249777 T1093 C513 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 249779 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@930fbbda[collection1] main{StandardDirectoryReader(segments_2:2:nrt)}
   [junit4]   2> 249780 T1093 C513 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 9
   [junit4]   2> 250402 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250426 T1106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_3,generation=3}
   [junit4]   2> 250427 T1106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 250447 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@9cfc8577[collection1] main
   [junit4]   2> 250448 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250449 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250450 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9cfc8577[collection1] main{StandardDirectoryReader(segments_3:14:nrt _3(4.8):C23/22:delGen=1 _2(4.8):C35/32:delGen=1)}
   [junit4]   2> 250478 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_3,generation=3}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_4,generation=4}
   [junit4]   2> 250478 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
   [junit4]   2> 250504 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@83e6d967[collection1] main
   [junit4]   2> 250505 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250505 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250506 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@83e6d967[collection1] main{StandardDirectoryReader(segments_4:26:nrt _6(4.8):C45/43:delGen=1 _7(4.8):C27/25:delGen=1)}
   [junit4]   2> 250520 T1105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_4,generation=4}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_5,generation=5}
   [junit4]   2> 250521 T1105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
   [junit4]   2> 250525 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@fc89441e[collection1] main
   [junit4]   2> 250526 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250526 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250527 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fc89441e[collection1] main{StandardDirectoryReader(segments_5:34:nrt _8(4.8):C18/16:delGen=2 _a(4.8):C2)}
   [junit4]   2> 250531 T1106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_5,generation=5}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_6,generation=6}
   [junit4]   2> 250531 T1106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
   [junit4]   2> 250534 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@abebca85[collection1] main
   [junit4]   2> 250535 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250535 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250536 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_6,generation=6}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_7,generation=7}
   [junit4]   2> 250536 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
   [junit4]   2> 250536 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@abebca85[collection1] main{StandardDirectoryReader(segments_6:40:nrt _a(4.8):C2/1:delGen=1 _b(4.8):C3/2:delGen=2 _c(4.8):C1)}
   [junit4]   2> 250537 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@9e490d0f[collection1] main
   [junit4]   2> 250538 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250538 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250539 T1105 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 250539 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9e490d0f[collection1] main{StandardDirectoryReader(segments_6:40:nrt _a(4.8):C2/1:delGen=1 _b(4.8):C3/2:delGen=2 _c(4.8):C1)}
   [junit4]   2> 250542 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@863708bc[collection1] main
   [junit4]   2> 250543 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250543 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250544 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@863708bc[collection1] main{StandardDirectoryReader(segments_7:45:nrt _a(4.8):C2/1:delGen=1 _b(4.8):C3/2:delGen=2 _d(4.8):C1)}
   [junit4]   2> 250547 T1106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_7,generation=7}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_8,generation=8}
   [junit4]   2> 250547 T1106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 8
   [junit4]   2> 250551 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@d50e7402[collection1] main
   [junit4]   2> 250551 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250552 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250552 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_8,generation=8}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_9,generation=9}
   [junit4]   2> 250552 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d50e7402[collection1] main{StandardDirectoryReader(segments_8:52:nrt _f(4.8):C1)}
   [junit4]   2> 250553 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 9
   [junit4]   2> 250554 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@c6dec7d7[collection1] main
   [junit4]   2> 250554 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250554 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250555 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c6dec7d7[collection1] main{StandardDirectoryReader(segments_8:52:nrt _f(4.8):C1)}
   [junit4]   2> 250557 T1105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_9,generation=9}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_a,generation=10}
   [junit4]   2> 250557 T1105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 10
   [junit4]   2> 250561 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@a39781cf[collection1] main
   [junit4]   2> 250562 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250562 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250563 T1106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_a,generation=10}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_b,generation=11}
   [junit4]   2> 250564 T1106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 11
   [junit4]   2> 250563 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a39781cf[collection1] main{StandardDirectoryReader(segments_a:58:nrt _f(4.8):C1 _h(4.8):C1)}
   [junit4]   2> 250564 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@42096bff[collection1] main
   [junit4]   2> 250565 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250565 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250567 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@42096bff[collection1] main{StandardDirectoryReader(segments_a:58:nrt _f(4.8):C1 _h(4.8):C1)}
   [junit4]   2> 250568 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_b,generation=11}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_c,generation=12}
   [junit4]   2> 250568 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 12
   [junit4]   2> 250624 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@e4d4f2bb[collection1] main
   [junit4]   2> 250625 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250627 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250628 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e4d4f2bb[collection1] main{StandardDirectoryReader(segments_c:64:nrt _f(4.8):C1 _j(4.8):C1)}
   [junit4]   2> 250629 T1105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_c,generation=12}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_d,generation=13}
   [junit4]   2> 250630 T1105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 13
   [junit4]   2> 250631 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@ef9ccd9b[collection1] main
   [junit4]   2> 250632 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250633 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250633 T1106 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 250635 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ef9ccd9b[collection1] main{StandardDirectoryReader(segments_c:64:nrt _f(4.8):C1 _j(4.8):C1)}
   [junit4]   2> 250639 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@e7b1ce25[collection1] main
   [junit4]   2> 250639 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250640 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250641 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e7b1ce25[collection1] main{StandardDirectoryReader(segments_d:69:nrt _f(4.8):C1 _k(4.8):C1)}
   [junit4]   2> 250644 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_d,generation=13}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_e,generation=14}
   [junit4]   2> 250644 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 14
   [junit4]   2> 250645 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@b8cfc339[collection1] main
   [junit4]   2> 250645 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250646 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250646 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b8cfc339[collection1] main{StandardDirectoryReader(segments_e:72:nrt _f(4.8):C1 _l(4.8):C1)}
   [junit4]   2> 250649 T1105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_e,generation=14}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_f,generation=15}
   [junit4]   2> 250649 T1105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 15
   [junit4]   2> 250652 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@873407d5[collection1] main
   [junit4]   2> 250653 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250653 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250654 T1106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_f,generation=15}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_g,generation=16}
   [junit4]   2> 250654 T1106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 16
   [junit4]   2> 250654 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@873407d5[collection1] main{StandardDirectoryReader(segments_f:80:nrt _l(4.8):C1 _n(4.8):C1)}
   [junit4]   2> 250655 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@29a8f3bf[collection1] main
   [junit4]   2> 250656 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250656 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250656 T1104 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 250657 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29a8f3bf[collection1] main{StandardDirectoryReader(segments_f:80:nrt _l(4.8):C1 _n(4.8):C1)}
   [junit4]   2> 250661 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@f85e9b53[collection1] main
   [junit4]   2> 250661 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250662 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250663 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f85e9b53[collection1] main{StandardDirectoryReader(segments_g:85:nrt _l(4.8):C1 _o(4.8):C1)}
   [junit4]   2> 250665 T1105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_g,generation=16}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_h,generation=17}
   [junit4]   2> 250665 T1105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 17
   [junit4]   2> 250666 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@c23173c2[collection1] main
   [junit4]   2> 250666 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250666 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250667 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c23173c2[collection1] main{StandardDirectoryReader(segments_h:88:nrt _l(4.8):C1 _p(4.8):C1)}
   [junit4]   2> 250669 T1106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_h,generation=17}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_i,generation=18}
   [junit4]   2> 250670 T1106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 18
   [junit4]   2> 250670 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@6ddb9f92[collection1] main
   [junit4]   2> 250671 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250671 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250672 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ddb9f92[collection1] main{StandardDirectoryReader(segments_i:92:nrt _l(4.8):C1 _q(4.8):C1)}
   [junit4]   2> 250675 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_i,generation=18}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_j,generation=19}
   [junit4]   2> 250675 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 19
   [junit4]   2> 250679 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@f50f4f3a[collection1] main
   [junit4]   2> 250679 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250680 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250681 T1105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_j,generation=19}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_k,generation=20}
   [junit4]   2> 250681 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f50f4f3a[collection1] main{StandardDirectoryReader(segments_j:100:nrt _r(4.8):C1 _s(4.8):C1)}
   [junit4]   2> 250682 T1105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 20
   [junit4]   2> 250685 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@1dcf72be[collection1] main
   [junit4]   2> 250686 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250686 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250687 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1dcf72be[collection1] main{StandardDirectoryReader(segments_k:102:nrt _t(4.8):C2)}
   [junit4]   2> 250741 T1106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_k,generation=20}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_l,generation=21}
   [junit4]   2> 250742 T1106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 21
   [junit4]   2> 250743 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@7e4172e9[collection1] main
   [junit4]   2> 250744 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250745 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250747 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e4172e9[collection1] main{StandardDirectoryReader(segments_l:105:nrt _t(4.8):C2/1:delGen=1 _u(4.8):C1)}
   [junit4]   2> 250751 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_l,generation=21}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_m,generation=22}
   [junit4]   2> 250752 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 22
   [junit4]   2> 250757 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@2870eec2[collection1] main
   [junit4]   2> 250757 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250758 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250759 T1105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_m,generation=22}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_n,generation=23}
   [junit4]   2> 250759 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2870eec2[collection1] main{StandardDirectoryReader(segments_m:111:nrt _t(4.8):C2/1:delGen=1 _v(4.8):C1 _w(4.8):C1)}
   [junit4]   2> 250760 T1105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 23
   [junit4]   2> 250761 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@c7c001e[collection1] main
   [junit4]   2> 250762 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250763 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250764 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c7c001e[collection1] main{StandardDirectoryReader(segments_m:111:nrt _t(4.8):C2/1:delGen=1 _v(4.8):C1 _w(4.8):C1)}
   [junit4]   2> 250769 T1106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_n,generation=23}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_o,generation=24}
   [junit4]   2> 250770 T1106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 24
   [junit4]   2> 250779 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@4ae60cdc[collection1] main
   [junit4]   2> 250780 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250780 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250781 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_o,generation=24}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_p,generation=25}
   [junit4]   2> 250782 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ae60cdc[collection1] main{StandardDirectoryReader(segments_o:119:nrt _v(4.8):C1 _y(4.8):C1)}
   [junit4]   2> 250782 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 25
   [junit4]   2> 250785 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@849569aa[collection1] main
   [junit4]   2> 250785 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250786 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250787 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@849569aa[collection1] main{StandardDirectoryReader(segments_p:121:nrt _z(4.8):c2)}
   [junit4]   2> 250802 T1105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_p,generation=25}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_q,generation=26}
   [junit4]   2> 250803 T1105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 26
   [junit4]   2> 250806 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@25619420[collection1] main
   [junit4]   2> 250807 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250808 T1106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250808 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@25619420[collection1] main{StandardDirectoryReader(segments_q:128:nrt _10(4.8):C1 _11(4.8):C1)}
   [junit4]   2> 250811 T1106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_q,generation=26}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_r,generation=27}
   [junit4]   2> 250812 T1106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 27
   [junit4]   2> 250814 T1106 oass.SolrIndexSearcher.<init> Opening Searcher@87712c8b[collection1] main
   [junit4]   2> 250815 T1106 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250815 T1104 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250816 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_r,generation=27}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_s,generation=28}
   [junit4]   2> 250816 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 28
   [junit4]   2> 250817 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@87712c8b[collection1] main{StandardDirectoryReader(segments_r:134:nrt _12(4.8):C1)}
   [junit4]   2> 250817 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@a5d62aa9[collection1] main
   [junit4]   2> 250818 T1104 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250818 T1105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 250819 T1105 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 250820 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a5d62aa9[collection1] main{StandardDirectoryReader(segments_r:134:nrt _12(4.8):C1)}
   [junit4]   2> 250820 T1105 oass.SolrIndexSearcher.<init> Opening Searcher@e2fd4d76[collection1] main
   [junit4]   2> 250821 T1105 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 250823 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e2fd4d76[collection1] main{StandardDirectoryReader(segments_r:134:nrt _12(4.8):C1)}
   [junit4]   2> ASYNC  NEW_CORE C514 name=collection1 org.apache.solr.core.SolrCore@13c78f74
   [junit4]   2> 251126 T1093 C514 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251126 T1093 C514 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 251127 T1093 C514 oass.SolrIndexSearcher.<init> Opening Searcher@4ca960b4[collection1] main
   [junit4]   2> 251128 T1093 C514 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251129 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ca960b4[collection1] main{StandardDirectoryReader(segments_r:134:nrt _12(4.8):C1)}
   [junit4]   2> 251130 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 306
   [junit4]   2> 251154 T1093 oas.SolrTestCaseJ4.tearDown ###Ending testMultiThreaded
   [junit4]   2> 251160 T1093 oas.SolrTestCaseJ4.setUp ###Starting testNonIndexedSignatureField
   [junit4]   2> 251160 T1093 C514 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 251161 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 251163 T1093 C514 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251163 T1093 C514 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_s,generation=28}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_t,generation=29}
   [junit4]   2> 251164 T1093 C514 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 29
   [junit4]   2> 251165 T1093 C514 oass.SolrIndexSearcher.<init> Opening Searcher@24dc7a58[collection1] main
   [junit4]   2> 251166 T1093 C514 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251167 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24dc7a58[collection1] main{StandardDirectoryReader(segments_t:135:nrt)}
   [junit4]   2> 251168 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 5
   [junit4]   2> 251173 T1093 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251176 T1093 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_t,generation=29}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_u,generation=30}
   [junit4]   2> 251177 T1093 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 30
   [junit4]   2> 251178 T1093 oass.SolrIndexSearcher.<init> Opening Searcher@f3f5b441[collection1] main
   [junit4]   2> 251179 T1093 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251180 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f3f5b441[collection1] main{StandardDirectoryReader(segments_u:137:nrt _13(4.8):C2)}
   [junit4]   2> 251181 T1093 oas.SolrTestCaseJ4.tearDown ###Ending testNonIndexedSignatureField
   [junit4]   2> 251186 T1093 oas.SolrTestCaseJ4.setUp ###Starting testDupeAllFieldsDetection
   [junit4]   2> 251186 T1093 C514 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 251187 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 251189 T1093 C514 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251190 T1093 C514 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_u,generation=30}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_v,generation=31}
   [junit4]   2> 251190 T1093 C514 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 31
   [junit4]   2> 251191 T1093 C514 oass.SolrIndexSearcher.<init> Opening Searcher@df24f79a[collection1] main
   [junit4]   2> 251192 T1093 C514 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251193 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@df24f79a[collection1] main{StandardDirectoryReader(segments_v:138:nrt)}
   [junit4]   2> 251194 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 5
   [junit4]   2> 251200 T1093 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251203 T1093 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_v,generation=31}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_w,generation=32}
   [junit4]   2> 251204 T1093 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 32
   [junit4]   2> 251205 T1093 oass.SolrIndexSearcher.<init> Opening Searcher@d8208753[collection1] main
   [junit4]   2> 251206 T1093 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251207 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d8208753[collection1] main{StandardDirectoryReader(segments_w:140:nrt _14(4.8):C3)}
   [junit4]   2> 251207 T1093 oas.SolrTestCaseJ4.tearDown ###Ending testDupeAllFieldsDetection
   [junit4]   2> 251212 T1093 oas.SolrTestCaseJ4.setUp ###Starting testDupeDetection
   [junit4]   2> 251213 T1093 C514 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 251213 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 0
   [junit4]   2> 251215 T1093 C514 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251216 T1093 C514 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_w,generation=32}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_x,generation=33}
   [junit4]   2> 251216 T1093 C514 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 33
   [junit4]   2> 251217 T1093 C514 oass.SolrIndexSearcher.<init> Opening Searcher@4cce4d82[collection1] main
   [junit4]   2> 251218 T1093 C514 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251219 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4cce4d82[collection1] main{StandardDirectoryReader(segments_x:141:nrt)}
   [junit4]   2> 251221 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 6
   [junit4]   2> 251226 T1093 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251229 T1093 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_x,generation=33}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_y,generation=34}
   [junit4]   2> 251229 T1093 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 34
   [junit4]   2> 251230 T1093 oass.SolrIndexSearcher.<init> Opening Searcher@eff73dd7[collection1] main
   [junit4]   2> 251231 T1093 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251232 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@eff73dd7[collection1] main{StandardDirectoryReader(segments_y:143:nrt _15(4.8):C2)}
   [junit4]   2> 251235 T1093 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251238 T1093 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_y,generation=34}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_z,generation=35}
   [junit4]   2> 251239 T1093 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 35
   [junit4]   2> 251240 T1093 oass.SolrIndexSearcher.<init> Opening Searcher@ad15f52c[collection1] main
   [junit4]   2> 251241 T1093 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251242 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ad15f52c[collection1] main{StandardDirectoryReader(segments_z:146:nrt _15(4.8):C2 _16(4.8):C1)}
   [junit4]   2> 251242 T1093 oas.SolrTestCaseJ4.tearDown ###Ending testDupeDetection
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SignatureUpdateProcessorFactoryTest -Dtests.method=testDupeDetection -Dtests.seed=1F78A35E78D2BB56 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mt_MT_PREEURO -Dtests.timezone=America/Grand_Turk -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 0.05s J1 | SignatureUpdateProcessorFactoryTest.testDupeDetection <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1> but was:<3>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1F78A35E78D2BB56:444858967B95182A]:0)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:78)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:126)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 251262 T1093 oas.SolrTestCaseJ4.setUp ###Starting testNonStringFieldsValues
   [junit4]   2> 251263 T1093 C514 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 251263 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 0
   [junit4]   2> 251265 T1093 C514 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251266 T1093 C514 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_z,generation=35}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_10,generation=36}
   [junit4]   2> 251266 T1093 C514 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 36
   [junit4]   2> 251267 T1093 C514 oass.SolrIndexSearcher.<init> Opening Searcher@1bffdf1e[collection1] main
   [junit4]   2> 251268 T1093 C514 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251269 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1bffdf1e[collection1] main{StandardDirectoryReader(segments_10:148:nrt)}
   [junit4]   2> 251270 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 5
   [junit4]   2> 251273 T1093 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 251275 T1093 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251278 T1093 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_10,generation=36}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_11,generation=37}
   [junit4]   2> 251279 T1093 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 37
   [junit4]   2> 251280 T1093 oass.SolrIndexSearcher.<init> Opening Searcher@81fa41bc[collection1] main
   [junit4]   2> 251281 T1093 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251282 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@81fa41bc[collection1] main{StandardDirectoryReader(segments_11:150:nrt _18(4.8):C5/1:delGen=1)}
   [junit4]   2> 251283 T1093 oas.SolrTestCaseJ4.tearDown ###Ending testNonStringFieldsValues
   [junit4]   2> 251288 T1093 oas.SolrTestCaseJ4.setUp ###Starting testFailNonIndexedSigWithOverwriteDupes
   [junit4]   2> 251289 T1093 C514 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 251290 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 251291 T1093 C514 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 251292 T1093 C514 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_11,generation=37}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@ac9df040 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50ddf94d)),segFN=segments_12,generation=38}
   [junit4]   2> 251293 T1093 C514 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 38
   [junit4]   2> 251294 T1093 C514 oass.SolrIndexSearcher.<init> Opening Searcher@993d0734[collection1] main
   [junit4]   2> 251294 T1093 C514 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 251296 T1095 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@993d0734[collection1] main{StandardDirectoryReader(segments_12:151:nrt)}
   [junit4]   2> 251297 T1093 C514 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 6
   [junit4]   2> 251298 T1093 oas.SolrTestCaseJ4.tearDown ###Ending testFailNonIndexedSigWithOverwriteDupes
   [junit4]   2> 251300 T1093 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 251300 T1093 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=613067616
   [junit4]   2> 251301 T1093 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@13c78f74
   [junit4]   2> 251310 T1093 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=42,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=253,cumulative_deletesById=0,cumulative_deletesByQuery=6,cumulative_errors=0}
   [junit4]   2> 251310 T1093 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 251311 T1093 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 251311 T1093 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 251312 T1093 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 251312 T1093 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 251313 T1093 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178/index;done=false>>]
   [junit4]   2> 251313 T1093 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178/index
   [junit4]   2> 251313 T1093 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178;done=false>>]
   [junit4]   2> 251314 T1093 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1393601212178
   [junit4]   2> NOTE: test params are: codec=Lucene46: {ints_is=FSTOrd41, weight=PostingsFormat(name=Lucene41WithOrds), name=PostingsFormat(name=Memory doPackFST= true), id=PostingsFormat(name=Memory doPackFST= true), v_t=PostingsFormat(name=Memory doPackFST= true), signatureField=PostingsFormat(name=MockSep)}, docValues:{}, sim=DefaultSimilarity, locale=mt_MT_PREEURO, timezone=America/Grand_Turk
   [junit4]   2> NOTE: Linux 3.8.0-36-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=147057440,total=239206400
   [junit4]   2> NOTE: All tests run in this JVM: [TestFastWriter, XsltUpdateRequestHandlerTest, DocValuesMultiTest, TestCSVResponseWriter, TestBinaryField, TestFoldingMultitermQuery, TestQueryTypes, TestCollationKeyRangeQueries, TestXIncludeConfig, UniqFieldsUpdateProcessorFactoryTest, CustomCollectionTest, TestNumberUtils, ConnectionManagerTest, HighlighterTest, OpenCloseCoreStressTest, MigrateRouteKeyTest, DisMaxRequestHandlerTest, TestIndexSearcher, RequestHandlersTest, ZkSolrClientTest, TestSerializedLuceneMatchVersion, HdfsCollectionsAPIDistributedZkTest, TestSchemaResource, TestPhraseSuggestions, TestFieldTypeCollectionResource, SuggesterWFSTTest, NumericFieldsTest, TestUpdate, SolrXmlInZkTest, TestSchemaNameResource, TestFuzzyAnalyzedSuggestions, TestDistributedSearch, CurrencyFieldOpenExchangeTest, SortByFunctionTest, TestFastOutputStream, TestCursorMarkWithoutUniqueKey, TestDFRSimilarityFactory, SignatureUpdateProcessorFactoryTest]
   [junit4] Completed on J1 in 2.56s, 6 tests, 1 failure <<< FAILURES!

[...truncated 40 lines...]
   [junit4] Suite: org.apache.solr.handler.component.TermVectorComponentDistributedTest
   [junit4]   2> 284493 T1355 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 284493 T1355 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 284498 T1355 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 284500 T1355 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 284504 T1355 oejus.SslContextFactory.doStart Enabled Protocols [TLSv1] of [SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 284511 T1355 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:54151
   [junit4]   2> 284513 T1355 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 284514 T1355 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 284515 T1355 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr
   [junit4]   2> 284515 T1355 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 284526 T1355 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 284587 T1355 oasc.CoreContainer.<init> New CoreContainer 804914544
   [junit4]   2> 284588 T1355 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 284589 T1355 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 284590 T1355 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 284591 T1355 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 284591 T1355 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 284592 T1355 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 284592 T1355 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 284593 T1355 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 284594 T1355 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 284594 T1355 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 284600 T1355 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 284601 T1355 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 284602 T1355 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 284605 T1365 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1
   [junit4]   2> 284605 T1365 oasc.SolrResourceLoader

[...truncated too long message...]

nit4]   2> 1129240 T4984 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1129243 T4984 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1129244 T4984 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1129245 T4984 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard1/data [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard1/data;done=false>>]
   [junit4]   2> 1129246 T4984 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard1/data
   [junit4]   2> 1129247 T4984 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard1/data/index [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard1/data/index;done=false>>]
   [junit4]   2> 1129247 T4984 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard1/data/index
   [junit4]   2> 1129250 T4984 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/zvcu/o,null}
   [junit4]   2> 1129323 T4984 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-877815459
   [junit4]   2> 1129324 T4984 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6f197ed5
   [junit4]   2> 1129330 T4984 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 1129330 T4984 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1129331 T4984 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1129332 T4984 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1129332 T4984 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1129333 T4984 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1129334 T4984 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard2/data/index [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard2/data/index;done=false>>]
   [junit4]   2> 1129336 T4984 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard2/data/index
   [junit4]   2> 1129336 T4984 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard2/data [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard2/data;done=false>>]
   [junit4]   2> 1129337 T4984 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1393602089987/shard2/data
   [junit4]   2> 1129338 T4984 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/zvcu/o,null}
   [junit4]   2> 1129390 T4984 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistributedQueryElevationComponentTest -Dtests.method=testDistribSearch -Dtests.seed=1F78A35E78D2BB56 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_MX -Dtests.timezone=Europe/Vienna -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 2.80s J1 | DistributedQueryElevationComponentTest.testDistribSearch <<<
   [junit4]    > Throwable #1: junit.framework.AssertionFailedError: .responseHeader.params.rows!=forceElevation (unordered or missing)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1F78A35E78D2BB56:9E9E2D460F8DDB6A]:0)
   [junit4]    > 	at junit.framework.Assert.fail(Assert.java:50)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
   [junit4]    > 	at org.apache.solr.handler.component.DistributedQueryElevationComponentTest.doTest(DistributedQueryElevationComponentTest.java:89)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 1129397 T4984 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=es_MX, timezone=Europe/Vienna
   [junit4]   2> NOTE: Linux 3.8.0-36-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=141588472,total=293666816
   [junit4]   2> NOTE: All tests run in this JVM: [TestFastWriter, XsltUpdateRequestHandlerTest, DocValuesMultiTest, TestCSVResponseWriter, TestBinaryField, TestFoldingMultitermQuery, TestQueryTypes, TestCollationKeyRangeQueries, TestXIncludeConfig, UniqFieldsUpdateProcessorFactoryTest, CustomCollectionTest, TestNumberUtils, ConnectionManagerTest, HighlighterTest, OpenCloseCoreStressTest, MigrateRouteKeyTest, DisMaxRequestHandlerTest, TestIndexSearcher, RequestHandlersTest, ZkSolrClientTest, TestSerializedLuceneMatchVersion, HdfsCollectionsAPIDistributedZkTest, TestSchemaResource, TestPhraseSuggestions, TestFieldTypeCollectionResource, SuggesterWFSTTest, NumericFieldsTest, TestUpdate, SolrXmlInZkTest, TestSchemaNameResource, TestFuzzyAnalyzedSuggestions, TestDistributedSearch, CurrencyFieldOpenExchangeTest, SortByFunctionTest, TestFastOutputStream, TestCursorMarkWithoutUniqueKey, TestDFRSimilarityFactory, SignatureUpdateProcessorFactoryTest, RequiredFieldsTest, TestSolrJ, TestSweetSpotSimilarityFactory, TestStressRecovery, MultiTermTest, BadCopyFieldTest, DistributedSuggestComponentTest, TestSolrDeletionPolicy2, HdfsBasicDistributedZk2Test, DistributedQueryComponentCustomSortTest, TestHighFrequencyDictionaryFactory, OutputWriterTest, TermVectorComponentDistributedTest, TriLevelCompositeIdRoutingTest, DefaultValueUpdateProcessorTest, PluginInfoTest, TestOverriddenPrefixQueryForCustomFieldType, QueryResultKeyTest, TestBinaryResponseWriter, RAMDirectoryFactoryTest, SolrCoreCheckLockOnStartupTest, AnalysisErrorHandlingTest, HighlighterConfigTest, TimeZoneUtilsTest, FieldAnalysisRequestHandlerTest, TestFiltering, TestJmxIntegration, HdfsLockFactoryTest, TestStressLucene, TestMultiCoreConfBootstrap, BasicDistributedZk2Test, TestSolr4Spatial, PrimUtilsTest, TestCodecSupport, TestComponentsName, TestOmitPositions, SpellCheckComponentTest, TestWordDelimiterFilterFactory, TestInitQParser, TestCollationField, TestUniqueKeyFieldResource, TestFastLRUCache, DocumentAnalysisRequestHandlerTest, CoreContainerCoreInitFailuresTest, FileUtilsTest, TestRecovery, JSONWriterTest, TestCSVLoader, AlternateDirectoryTest, ChaosMonkeyNothingIsSafeTest, BasicDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, ZkCLITest, ShardRoutingCustomTest, SliceStateUpdateTest, TestDistributedGrouping, DistributedSpellCheckComponentTest, TestJoin, TestRangeQuery, StatsComponentTest, QueryElevationComponentTest, BadIndexSchemaTest, TestSort, DirectUpdateHandlerTest, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest]
   [junit4] Completed on J1 in 2.82s, 1 test, 1 failure <<< FAILURES!

[...truncated 437 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:464: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1278: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:911: There were test failures: 363 suites, 1570 tests, 3 failures, 36 ignored (8 assumptions)

Total time: 52 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-fcs-b129) - Build # 9520 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9520/
Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ShardSplitTest.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:47455

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:47455
	at __randomizedtesting.SeedInfo.seed([B21111266F3E552:8AC79F0A11AC856E]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:507)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:536)
	at org.apache.solr.cloud.ShardSplitTest.incompleteOrOverlappingCustomRangeTest(ShardSplitTest.java:159)
	at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:111)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:150)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
	... 45 more




Build Log:
[...truncated 10275 lines...]
   [junit4] Suite: org.apache.solr.cloud.ShardSplitTest
   [junit4]   2> 16851 T48 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 16864 T48 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-ShardSplitTest-1393614572244
   [junit4]   2> 16867 T48 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 16869 T49 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 16969 T48 oasc.ZkTestServer.run start zk server on port:42702
   [junit4]   2> 17030 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 17046 T50 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 18597 T55 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@293765 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 18598 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 18599 T48 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 18626 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 18628 T57 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16b7080 name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 18628 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 18634 T48 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 18638 T48 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 18641 T48 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 18644 T48 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 18647 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 18657 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 18662 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 18663 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 18668 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 18669 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 18672 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 18673 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 18676 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 18676 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 18680 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 18680 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 18683 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 18684 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 18687 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 18687 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 18691 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 18691 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 18694 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 18695 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 18698 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 18699 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 19081 T48 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 19161 T48 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41673
   [junit4]   2> 19201 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 19202 T48 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19202 T48 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084
   [junit4]   2> 19202 T48 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/'
   [junit4]   2> 19212 T48 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/solr.xml
   [junit4]   2> 19224 T48 oasc.CoreContainer.<init> New CoreContainer 1824674
   [junit4]   2> 19224 T48 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/]
   [junit4]   2> 19234 T48 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 19235 T48 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 19235 T48 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 19235 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 19236 T48 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 19236 T48 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 19236 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 19236 T48 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 19237 T48 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 19239 T48 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 19239 T48 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 19240 T48 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 19240 T48 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42702/solr
   [junit4]   2> 19249 T48 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 19250 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19253 T69 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1158d81 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19254 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19262 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19266 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48654d name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19266 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19274 T48 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 19283 T48 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 19294 T48 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 19297 T48 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41673_
   [junit4]   2> 19301 T48 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41673_
   [junit4]   2> 19307 T48 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 19309 T48 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 19318 T48 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41673_
   [junit4]   2> 19318 T48 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 19321 T48 oasc.Overseer.start Overseer (id=91331924614250499-127.0.0.1:41673_-n_0000000000) starting
   [junit4]   2> 19331 T48 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 19344 T73 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 19344 T48 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 19347 T48 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 19351 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 19353 T73 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 19360 T72 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 19365 T74 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 19365 T74 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 19371 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19371 T74 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 19374 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 19375 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41673",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41673_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 19375 T72 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 19382 T72 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 19388 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19392 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 20372 T74 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 20372 T74 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1
   [junit4]   2> 20372 T74 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 20373 T74 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 20374 T74 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 20375 T74 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 20375 T74 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/'
   [junit4]   2> 20376 T74 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/lib/.svn/' to classloader
   [junit4]   2> 20376 T74 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/lib/classes/' to classloader
   [junit4]   2> 20377 T74 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/lib/README' to classloader
   [junit4]   2> 20409 T74 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 20423 T74 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 20425 T74 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 20433 T74 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 20624 T74 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 20625 T74 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 20626 T74 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 20629 T74 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 20660 T74 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 20661 T74 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data/
   [junit4]   2> 20661 T74 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@142865a
   [junit4]   2> 20665 T74 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data
   [junit4]   2> 20666 T74 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data/index/
   [junit4]   2> 20666 T74 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 20666 T74 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data/index
   [junit4]   2> 20667 T74 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7989922721774966]
   [junit4]   2> 20667 T74 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@101186d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad24e0),segFN=segments_1,generation=1}
   [junit4]   2> 20668 T74 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 20673 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 20684 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 20684 T74 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 20685 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 20685 T74 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 20685 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 20685 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 20686 T74 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 20686 T74 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 20688 T74 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 20689 T74 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 20689 T74 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 20689 T74 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 20690 T74 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 20690 T74 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 20691 T74 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 20693 T74 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 20694 T74 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 20694 T74 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 20696 T74 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=9.20703125, floorSegmentMB=1.8662109375, forceMergeDeletesPctAllowed=5.930420212113585, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4004882394888162
   [junit4]   2> 20696 T74 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@101186d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad24e0),segFN=segments_1,generation=1}
   [junit4]   2> 20697 T74 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 20697 T74 oass.SolrIndexSearcher.<init> Opening Searcher@19e5fa5[collection1] main
   [junit4]   2> 20703 T75 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19e5fa5[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 20707 T74 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 20708 T78 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41673 collection:control_collection shard:shard1
   [junit4]   2> 20708 T48 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 20708 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 20715 T78 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 20723 T78 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 20727 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 20728 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20729 T78 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 20729 T80 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14dab45 name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 20729 T78 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 20729 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> ASYNC  NEW_CORE C40 name=collection1 org.apache.solr.core.SolrCore@872f43 url=http://127.0.0.1:41673/collection1 node=127.0.0.1:41673_ C40_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:41673, core=collection1, node_name=127.0.0.1:41673_}
   [junit4]   2> 20730 T78 C40 P41673 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41673/collection1/
   [junit4]   2> 20730 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 20731 T78 C40 P41673 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 20731 T78 C40 P41673 oasc.SyncStrategy.syncToMe http://127.0.0.1:41673/collection1/ has no replicas
   [junit4]   2> 20731 T78 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41673/collection1/ shard1
   [junit4]   2> 20731 T78 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 20733 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 20734 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20741 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20744 T48 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 20749 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20853 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 20856 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 20893 T78 oasc.ZkController.register We are http://127.0.0.1:41673/collection1/ and leader is http://127.0.0.1:41673/collection1/
   [junit4]   2> 20894 T78 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41673
   [junit4]   2> 20894 T78 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 20894 T78 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 20894 T78 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 20895 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20895 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20896 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20896 T78 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 20897 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 20898 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41673",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41673_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 20901 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20995 T48 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 20996 T48 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 20999 T48 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36156
   [junit4]   2> 21000 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 21000 T48 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 21000 T48 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124
   [junit4]   2> 21000 T48 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/'
   [junit4]   2> 21003 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 21003 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 21010 T48 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/solr.xml
   [junit4]   2> 21025 T48 oasc.CoreContainer.<init> New CoreContainer 22746881
   [junit4]   2> 21026 T48 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/]
   [junit4]   2> 21026 T48 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 21027 T48 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 21027 T48 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 21027 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 21028 T48 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 21028 T48 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 21028 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 21029 T48 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 21029 T48 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 21032 T48 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 21032 T48 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 21032 T48 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 21033 T48 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42702/solr
   [junit4]   2> 21033 T48 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 21034 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21038 T91 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15d96c0 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21039 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21043 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21045 T93 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@796175 name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21045 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21049 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 22052 T48 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36156_
   [junit4]   2> 22054 T48 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36156_
   [junit4]   2> 22057 T71 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 22057 T93 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 22057 T80 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 22062 T94 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 22062 T94 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 22063 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22063 T94 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 22063 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22064 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22065 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 22066 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36156",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36156_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 22066 T72 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 22066 T72 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 22068 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22173 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22173 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22173 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23064 T94 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 23064 T94 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1
   [junit4]   2> 23064 T94 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 23065 T94 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 23065 T94 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 23068 T94 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 23068 T94 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/'
   [junit4]   2> 23069 T94 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/lib/.svn/' to classloader
   [junit4]   2> 23069 T94 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/lib/classes/' to classloader
   [junit4]   2> 23070 T94 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/lib/README' to classloader
   [junit4]   2> 23100 T94 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 23116 T94 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 23118 T94 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 23125 T94 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 23305 T94 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 23307 T94 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 23308 T94 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 23313 T94 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 23365 T94 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 23366 T94 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1/
   [junit4]   2> 23366 T94 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@142865a
   [junit4]   2> 23367 T94 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1
   [junit4]   2> 23367 T94 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1/index/
   [junit4]   2> 23367 T94 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 23368 T94 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1/index
   [junit4]   2> 23368 T94 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7989922721774966]
   [junit4]   2> 23369 T94 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2694cf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@694c7),segFN=segments_1,generation=1}
   [junit4]   2> 23370 T94 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 23377 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 23378 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 23378 T94 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 23378 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 23378 T94 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 23379 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 23383 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 23384 T94 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 23384 T94 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 23388 T94 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 23389 T94 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 23389 T94 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 23397 T94 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 23398 T94 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 23399 T94 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 23400 T94 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 23403 T94 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 23405 T94 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 23405 T94 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 23406 T94 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=9.20703125, floorSegmentMB=1.8662109375, forceMergeDeletesPctAllowed=5.930420212113585, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4004882394888162
   [junit4]   2> 23407 T94 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2694cf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@694c7),segFN=segments_1,generation=1}
   [junit4]   2> 23407 T94 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 23408 T94 oass.SolrIndexSearcher.<init> Opening Searcher@892d63[collection1] main
   [junit4]   2> 23421 T95 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@892d63[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 23423 T94 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 23424 T48 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 23425 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 23426 T98 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36156 collection:collection1 shard:shard2
   [junit4]   2> 23428 T98 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 23444 T98 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 23450 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23450 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23451 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23451 T98 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 23451 T98 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C41 name=collection1 org.apache.solr.core.SolrCore@15ff89f url=http://127.0.0.1:36156/collection1 node=127.0.0.1:36156_ C41_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:36156, core=collection1, node_name=127.0.0.1:36156_}
   [junit4]   2> 23451 T98 C41 P36156 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36156/collection1/
   [junit4]   2> 23452 T98 C41 P36156 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 23452 T98 C41 P36156 oasc.SyncStrategy.syncToMe http://127.0.0.1:36156/collection1/ has no replicas
   [junit4]   2> 23452 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 23453 T98 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36156/collection1/ shard2
   [junit4]   2> 23453 T98 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 23458 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23462 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23466 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23568 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23568 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23568 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23613 T98 oasc.ZkController.register We are http://127.0.0.1:36156/collection1/ and leader is http://127.0.0.1:36156/collection1/
   [junit4]   2> 23613 T98 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36156
   [junit4]   2> 23613 T98 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 23613 T98 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 23613 T98 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 23615 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23615 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23615 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23615 T98 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 23617 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 23618 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36156",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36156_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 23629 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23731 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23731 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23731 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23756 T48 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 23756 T48 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 23758 T48 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47455
   [junit4]   2> 23759 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 23760 T48 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 23760 T48 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805
   [junit4]   2> 23760 T48 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/'
   [junit4]   2> 23770 T48 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/solr.xml
   [junit4]   2> 23784 T48 oasc.CoreContainer.<init> New CoreContainer 6099137
   [junit4]   2> 23785 T48 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/]
   [junit4]   2> 23785 T48 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 23786 T48 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 23786 T48 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 23786 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 23787 T48 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 23787 T48 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 23788 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 23788 T48 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 23788 T48 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 23791 T48 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 23791 T48 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 23791 T48 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 23791 T48 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42702/solr
   [junit4]   2> 23792 T48 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 23793 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23795 T109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11a85b4 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23795 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23797 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23799 T111 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d698b name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23799 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23822 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 24826 T48 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47455_
   [junit4]   2> 24828 T48 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47455_
   [junit4]   2> 24831 T93 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 24831 T71 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 24832 T111 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 24832 T80 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 24842 T112 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 24842 T112 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 24845 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24845 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24845 T112 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 24846 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24847 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 24848 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47455",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47455_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 24848 T72 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 24849 T72 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 24851 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24955 T111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24955 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24955 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24955 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 25846 T112 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 25846 T112 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1
   [junit4]   2> 25846 T112 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 25847 T112 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 25847 T112 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 25848 T112 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 25848 T112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/'
   [junit4]   2> 25849 T112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/lib/.svn/' to classloader
   [junit4]   2> 25849 T112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/lib/classes/' to classloader
   [junit4]   2> 25849 T112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/lib/README' to classloader
   [junit4]   2> 25876 T112 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 25895 T112 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 25896 T112 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 25903 T112 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 26045 T112 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 26047 T112 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 26048 T112 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 26051 T112 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 26075 T112 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 26075 T112 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2/
   [junit4]   2> 26075 T112 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@142865a
   [junit4]   2> 26076 T112 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2
   [junit4]   2> 26077 T112 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2/index/
   [junit4]   2> 26077 T112 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 26078 T112 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2/index
   [junit4]   2> 26078 T112 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7989922721774966]
   [junit4]   2> 26079 T112 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a4f837 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@27e451),segFN=segments_1,generation=1}
   [junit4]   2> 26079 T112 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26086 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 26087 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 26087 T112 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 26087 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 26088 T112 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 26088 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 26088 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 26089 T112 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 26089 T112 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 26090 T112 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 26091 T112 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 26091 T112 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 26091 T112 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 26092 T112 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 26093 T112 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 26094 T112 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 26096 T112 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26098 T112 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 26098 T112 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 26100 T112 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=9.20703125, floorSegmentMB=1.8662109375, forceMergeDeletesPctAllowed=5.930420212113585, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4004882394888162
   [junit4]   2> 26101 T112 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a4f837 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@27e451),segFN=segments_1,generation=1}
   [junit4]   2> 26101 T112 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26102 T112 oass.SolrIndexSearcher.<init> Opening Searcher@fd4727[collection1] main
   [junit4]   2> 26108 T113 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fd4727[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 26112 T112 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 26113 T48 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 26113 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 26113 T116 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47455 collection:collection1 shard:shard1
   [junit4]   2> 26115 T116 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 26121 T116 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 26123 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26123 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26123 T116 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 26124 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26124 T116 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C42 name=collection1 org.apache.solr.core.SolrCore@12d4227 url=http://127.0.0.1:47455/collection1 node=127.0.0.1:47455_ C42_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:47455, core=collection1, node_name=127.0.0.1:47455_}
   [junit4]   2> 26124 T116 C42 P47455 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47455/collection1/
   [junit4]   2> 26125 T116 C42 P47455 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 26125 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26126 T116 C42 P47455 oasc.SyncStrategy.syncToMe http://127.0.0.1:47455/collection1/ has no replicas
   [junit4]   2> 26126 T116 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47455/collection1/ shard1
   [junit4]   2> 26127 T116 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 26130 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26134 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26138 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26240 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26240 T111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26240 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26240 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26285 T116 oasc.ZkController.register We are http://127.0.0.1:47455/collection1/ and leader is http://127.0.0.1:47455/collection1/
   [junit4]   2> 26285 T116 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47455
   [junit4]   2> 26285 T116 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 26285 T116 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 26286 T116 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 26287 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26287 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26287 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26287 T116 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26288 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26289 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47455",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47455_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 26294 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26339 T48 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 26340 T48 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 26342 T48 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51770
   [junit4]   2> 26343 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 26344 T48 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 26344 T48 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494
   [junit4]   2> 26344 T48 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/'
   [junit4]   2> 26355 T48 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/solr.xml
   [junit4]   2> 26369 T48 oasc.CoreContainer.<init> New CoreContainer 26212847
   [junit4]   2> 26370 T48 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/]
   [junit4]   2> 26370 T48 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 26371 T48 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 26371 T48 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 26371 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 26372 T48 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 26372 T48 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 26372 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 26372 T48 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 26373 T48 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 26374 T48 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 26375 T48 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 26375 T48 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 26375 T48 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42702/solr
   [junit4]   2> 26376 T48 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 26376 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26378 T127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b263e3 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26379 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26382 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26383 T129 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cf373 name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26383 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26387 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 26397 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26397 T129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26397 T111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26397 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26397 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 27390 T48 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51770_
   [junit4]   2> 27392 T48 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51770_
   [junit4]   2> 27396 T111 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27397 T80 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27397 T129 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27397 T93 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27397 T71 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27408 T130 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 27409 T130 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 27410 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27410 T130 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 27410 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27411 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27412 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 27413 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51770",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51770_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 27413 T72 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 27413 T72 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 27416 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27518 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 27519 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 27518 T111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 27518 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 27518 T129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 28411 T130 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 28411 T130 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1
   [junit4]   2> 28411 T130 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 28412 T130 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 28412 T130 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 28414 T130 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 28414 T130 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/'
   [junit4]   2> 28414 T130 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/lib/.svn/' to classloader
   [junit4]   2> 28415 T130 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/lib/classes/' to classloader
   [junit4]   2> 28415 T130 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/lib/README' to classloader
   [junit4]   2> 28449 T130 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 28464 T130 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 28465 T130 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 28470 T130 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 28623 T130 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 28625 T130 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 28625 T130 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 28628 T130 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 28650 T130 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 28650 T130 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3/
   [junit4]   2> 28650 T130 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@142865a
   [junit4]   2> 28651 T130 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3
   [junit4]   2> 28652 T130 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3/index/
   [junit4]   2> 28652 T130 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 28653 T130 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3/index
   [junit4]   2> 28654 T130 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7989922721774966]
   [junit4]   2> 28655 T130 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@e14509 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ab8da),segFN=segments_1,generation=1}
   [junit4]   2> 28656 T130 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 28661 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 28662 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 28662 T130 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 28662 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 28662 T130 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 28663 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 28663 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 28663 T130 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 28664 T130 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 28664 T130 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 28665 T130 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 28665 T130 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]

[...truncated too long message...]

Elector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> ASYNC  NEW_CORE C52 name=collection1 org.apache.solr.core.SolrCore@1c0703b url=http://127.0.0.1:40234/collection1 node=127.0.0.1:40234_ C52_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:40234, core=collection1, node_name=127.0.0.1:40234_}
   [junit4]   2> 340333 T148 C52 P40234 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 340334 T148 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 340334 T148 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40234/collection1/ shard1
   [junit4]   2> 340334 T148 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1c0703b
   [junit4]   2> 340341 T148 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 340342 T148 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 340342 T148 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 340342 T148 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 340342 T148 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4;done=false>>]
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4/index;done=false>>]
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4/index
   [junit4]   2> 340343 T148 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 340344 T148 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:414)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 340344 T148 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election/91331924614250508-core_node4-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 340345 T148 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 340345 T148 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=lv, timezone=SystemV/PST8PDT
   [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=1,free=12743496,total=64880640
   [junit4]   2> NOTE: All tests run in this JVM: [DocValuesTest, TestSurroundQueryParser, IndexSchemaTest, TestTrie, TestFunctionQuery, ReturnFieldsTest, TestMaxScoreQueryParser, ShardSplitTest]
   [junit4] Completed on J0 in 324.41s, 1 test, 1 error <<< FAILURES!

[...truncated 966 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:451: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1278: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:911: There were test failures: 363 suites, 1570 tests, 1 error, 36 ignored (8 assumptions)

Total time: 51 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_51) - Build # 9519 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9519/
Java: 64bit/jdk1.7.0_51 -XX:+UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord

1 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([6326A836295DCF79:8050A1A3CE275A0B]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
	... 47 more




Build Log:
[...truncated 11597 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> 10483 T50 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 10484 T50 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /sm_gby/a
   [junit4]   2> 10486 T50 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 10504 T50 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393608903188
   [junit4]   2> 10508 T50 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 10511 T51 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 10657 T50 oasc.ZkTestServer.run start zk server on port:39068
   [junit4]   2> 10690 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 10700 T52 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 45257 T54 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 32939ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 45280 T52 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14479904a4b0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 45282 T52 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14479904a4b0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 45285 T57 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71284e3 name:ZooKeeperConnection Watcher:127.0.0.1:39068 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 45286 T57 oascc.ConnectionManager.process Client->ZooKeeper status change trigger but we are already closed
   [junit4]   2> 45292 T50 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 45296 T50 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39068 39068
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testShutdown -Dtests.seed=6326A836295DCF79 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=vi_VN -Dtests.timezone=Pacific/Marquesas -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   40.0s J0 | CloudSolrServerTest.testShutdown <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6326A836295DCF79:8050A1A3CE275A0B]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
   [junit4]    > 	... 47 more
   [junit4]   2> 50513 T50 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393608943196
   [junit4]   2> 50513 T50 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 50514 T58 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 50614 T50 oasc.ZkTestServer.run start zk server on port:49340
   [junit4]   2> 50615 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50618 T64 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@378f5b14 name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50618 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50620 T50 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 50642 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50643 T66 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@330fd2da name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50644 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50648 T50 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 50653 T50 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 50659 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 50669 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 50674 T50 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 50685 T50 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 50691 T50 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 50692 T50 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 50702 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 50703 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 50703 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 50703 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 50704 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 50704 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 50705 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 50705 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 50705 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 50763 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 50792 T50 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33205
   [junit4]   2> 50793 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 50793 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 50793 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402
   [junit4]   2> 50794 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/'
   [junit4]   2> 50806 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/solr.xml
   [junit4]   2> 50858 T50 oasc.CoreContainer.<init> New CoreContainer 1700712206
   [junit4]   2> 50858 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/]
   [junit4]   2> 50859 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 50860 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 50860 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 50861 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 50861 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 50862 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 50862 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 50863 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 50864 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 50868 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 50868 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 50869 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 50869 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49340/solr
   [junit4]   2> 50882 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 50883 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50887 T77 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a22c38f name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50888 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50893 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50896 T79 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74845058 name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50896 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50901 T50 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 50909 T50 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 50919 T50 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 50922 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33205_sm_gby%2Fa
   [junit4]   2> 50926 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33205_sm_gby%2Fa
   [junit4]   2> 50932 T50 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 50936 T50 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 50946 T50 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:33205_sm_gby%2Fa
   [junit4]   2> 50947 T50 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 50951 T50 oasc.Overseer.start Overseer (id=91331555701555203-127.0.0.1:33205_sm_gby%2Fa-n_0000000000) starting
   [junit4]   2> 50964 T50 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 51018 T81 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 51019 T50 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 51023 T50 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 51026 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 51028 T81 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 51035 T80 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 51042 T82 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 51043 T82 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 51046 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 51047 T82 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 51050 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 51052 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33205/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33205_sm_gby%2Fa",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 51052 T80 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 51059 T80 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 51064 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 51068 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52047 T82 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 52048 T82 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/collection1
   [junit4]   2> 52049 T82 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 52050 T82 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 52050 T82 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 52053 T82 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 52053 T82 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/collection1/'
   [junit4]   2> 52113 T82 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 52166 T82 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 52169 T82 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 52230 T82 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 52301 T82 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 52311 T82 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 52321 T82 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 52331 T82 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 52709 T82 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 52712 T82 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 52715 T82 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 52715 T82 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 52720 T82 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 52720 T82 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/
   [junit4]   2> 52720 T82 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 52721 T82 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data
   [junit4]   2> 52721 T82 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index/
   [junit4]   2> 52722 T82 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 52722 T82 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index
   [junit4]   2> 52723 T82 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@52d6c570 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 52724 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 52725 T82 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 52725 T82 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 52725 T82 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 52726 T82 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 52729 T82 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 52730 T82 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 52730 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 52730 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 52732 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 52735 T82 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 52735 T82 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 52736 T82 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@52d6c570 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 52737 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 52737 T82 oass.SolrIndexSearcher.<init> Opening Searcher@614b722[collection1] main
   [junit4]   2> 52738 T83 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@614b722[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 52738 T82 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 52740 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 52741 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 52740 T86 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33205/sm_gby/a collection:control_collection shard:shard1
   [junit4]   2> 52747 T86 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 52749 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52751 T88 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29f8f02c name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52751 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52756 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 52769 T86 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 52771 T50 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 52777 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 52784 T86 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 52784 T86 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 52785 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@488b250 url=http://127.0.0.1:33205/sm_gby/a/collection1 node=127.0.0.1:33205_sm_gby%2Fa C6_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:33205/sm_gby/a, core=collection1, node_name=127.0.0.1:33205_sm_gby%2Fa}
   [junit4]   2> 52786 T86 C6 P33205 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33205/sm_gby/a/collection1/
   [junit4]   2> 52787 T86 C6 P33205 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 52787 T86 C6 P33205 oasc.SyncStrategy.syncToMe http://127.0.0.1:33205/sm_gby/a/collection1/ has no replicas
   [junit4]   2> 52787 T86 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33205/sm_gby/a/collection1/ shard1
   [junit4]   2> 52788 T86 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 52793 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 52794 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52794 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52810 T50 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 52810 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 52813 T50 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54864
   [junit4]   2> 52814 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 52814 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 52815 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454
   [junit4]   2> 52815 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/'
   [junit4]   2> 52827 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/solr.xml
   [junit4]   2> 52873 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 52874 T50 oasc.CoreContainer.<init> New CoreContainer 1597724868
   [junit4]   2> 52875 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/]
   [junit4]   2> 52876 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 52876 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 52877 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 52877 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 52877 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 52878 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 52878 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 52878 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 52879 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 52880 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 52884 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 52885 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 52885 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 52886 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49340/solr
   [junit4]   2> 52886 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 52888 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52892 T99 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d9e8821 name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52892 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52896 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52899 T101 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ede9641 name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52899 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52906 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 52984 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52984 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52984 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 53023 T86 oasc.ZkController.register We are http://127.0.0.1:33205/sm_gby/a/collection1/ and leader is http://127.0.0.1:33205/sm_gby/a/collection1/
   [junit4]   2> 53023 T86 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33205/sm_gby/a
   [junit4]   2> 53023 T86 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 53024 T86 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 53024 T86 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 53025 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53026 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53026 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53026 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 53028 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 53030 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33205/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33205_sm_gby%2Fa",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 53034 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53137 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 53137 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 53138 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 53911 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54864_sm_gby%2Fa
   [junit4]   2> 53913 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54864_sm_gby%2Fa
   [junit4]   2> 53918 T101 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 53918 T79 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 53919 T88 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 53934 T102 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 53934 T102 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 53937 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53937 T102 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 53937 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53938 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53940 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 53942 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54864/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54864_sm_gby%2Fa",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 53942 T80 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 53943 T80 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 53949 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 54053 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 54053 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 54054 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 54938 T102 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 54939 T102 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/collection1
   [junit4]   2> 54939 T102 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 54940 T102 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 54940 T102 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 54942 T102 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 54942 T102 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/collection1/'
   [junit4]   2> 54976 T102 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 55018 T102 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 55021 T102 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 55070 T102 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 55453 T102 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 55456 T102 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 55458 T102 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 55459 T102 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 55460 T102 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 55460 T102 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/
   [junit4]   2> 55461 T102 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 55461 T102 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1
   [junit4]   2> 55462 T102 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index/
   [junit4]   2> 55462 T102 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 55462 T102 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index
   [junit4]   2> 55464 T102 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@38516a27 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 55464 T102 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 55465 T102 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 55465 T102 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 55466 T102 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 55466 T102 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 55466 T102 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 55467 T102 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 55467 T102 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 55467 T102 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 55469 T102 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 55472 T102 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 55473 T102 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 55474 T102 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@38516a27 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 55474 T102 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 55474 T102 oass.SolrIndexSearcher.<init> Opening Searcher@5421641[collection1] main
   [junit4]   2> 55475 T103 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5421641[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 55475 T102 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 55476 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 55477 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 55476 T106 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54864/sm_gby/a collection:collection1 shard:shard2
   [junit4]   2> 55479 T106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 55490 T106 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 55493 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55493 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55494 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55494 T106 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 55494 T106 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@4c112d16 url=http://127.0.0.1:54864/sm_gby/a/collection1 node=127.0.0.1:54864_sm_gby%2Fa C7_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:54864/sm_gby/a, core=collection1, node_name=127.0.0.1:54864_sm_gby%2Fa}
   [junit4]   2> 55495 T106 C7 P54864 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54864/sm_gby/a/collection1/
   [junit4]   2> 55495 T106 C7 P54864 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 55495 T106 C7 P54864 oasc.SyncStrategy.syncToMe http://127.0.0.1:54864/sm_gby/a/collection1/ has no replicas
   [junit4]   2> 55495 T106 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54864/sm_gby/a/collection1/ shard2
   [junit4]   2> 55496 T106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 55496 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 55501 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55506 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55508 T50 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 55508 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 55511 T50 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49022
   [junit4]   2> 55512 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55512 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 55513 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 55513 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160
   [junit4]   2> 55513 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/'
   [junit4]   2> 55526 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/solr.xml
   [junit4]   2> 55581 T50 oasc.CoreContainer.<init> New CoreContainer 853927167
   [junit4]   2> 55581 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/]
   [junit4]   2> 55583 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 55583 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 55584 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 55584 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 55585 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 55585 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 55586 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 55586 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 55587 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 55590 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 55591 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 55591 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 55591 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49340/solr
   [junit4]   2> 55592 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 55592 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 55594 T117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@131d3b74 name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 55595 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 55598 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 55599 T119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70c18d7f name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 55600 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 55606 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 55646 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55646 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55646 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55646 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55657 T106 oasc.ZkController.register We are http://127.0.0.1:54864/sm_gby/a/collection1/ and leader is http://127.0.0.1:54864/sm_gby/a/collection1/
   [junit4]   2> 55657 T106 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54864/sm_gby/a
   [junit4]   2> 55658 T106 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 55658 T106 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 55658 T106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 55661 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55661 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55661 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55661 T106 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 55663 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 55665 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54864/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54864_sm_gby%2Fa",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 55669 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55772 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55772 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55772 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55772 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 56612 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49022_sm_gby%2Fa
   [junit4]   2> 56615 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49022_sm_gby%2Fa
   [junit4]   2> 56622 T88 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 56622 T79 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 56623 T119 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 56624 T101 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 56643 T120 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 56643 T120 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 56646 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 56646 T120 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 56646 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 56647 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 56652 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 56655 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49022/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49022_sm_gby%2Fa",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 56656 T80 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 56656 T80 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 56663 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 56769 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 56769 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 56769 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 56769 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 57647 T120 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 57648 T120 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/collection1
   [junit4]   2> 57648 T120 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 57649 T120 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 57649 T120 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 57651 T120 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 57651 T120 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/collection1/'
   [junit4]   2> 57700 T120 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 57739 T120 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 57741 T120 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 57770 T120 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 58148 T120 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 58151 T120 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 58153 T120 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 58153 T120 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 58154 T120 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 58155 T120 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/
   [junit4]   2> 58155 T120 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 58156 T120 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2
   [junit4]   2> 58156 T120 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index/
   [junit4]   2> 58156 T120 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 58156 T120 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index
   [junit4]   2> 58158 T120 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2c82719b lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 58158 T120 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 58159 T120 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 58159 T120 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 58160 T120 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 58160 T120 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 58160 T120 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 58161 T120 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 58161 T120 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 58161 T120 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 58163 T120 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 58166 T120 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 58167 T120 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 58167 T120 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2c82719b lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 58168 T120 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 58168 T120 oass.SolrIndexSearcher.<init> Opening Searcher@79c1bab1[collection1] main
   [junit4]   2> 58169 T121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79c1bab1[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 58169 T120 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 58171 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 58171 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 58171 T124 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49022/sm_gby/a collection:collection1 shard:shard1
   [junit4]   2> 58173 T124 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 58181 T124 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 58184 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58185 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58185 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58186 T124 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 58186 T124 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8 name=collection1 org.apache.solr.core.SolrCore@66c47c26 url=http://127.0.0.1:49022/sm_gby/a/collection1 node=127.0.0.1:49022_sm_gby%2Fa C8_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:49022/sm_gby/a, core=collection1, node_name=127.0.0.1:49022_sm_gby%2Fa}
   [junit4]   2> 58186 T124 C8 P49022 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49022/sm_gby/a/collection1/
   [junit4]   2> 58188 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 58188 T124 C8 P49022 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 58189 T124 C8 P49022 oasc.SyncStrategy.syncToMe http://127.0.0.1:49022/sm_gby/a/collection1/ has no replicas
   [junit4]   2> 58190 T124 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49022/sm_gby/a/collection1/ shard1
   [junit4]   2> 58190 T124 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 58193 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58201 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58203 T50 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 58204 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 58206 T50 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52027
   [junit4]   2> 58207 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 58208 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58208 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 58209 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855
   [junit4]   2> 58209 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/'
   [junit4]   2> 58229 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/solr.xml
   [junit4]   2> 58273 T50 oasc.CoreContainer.<init> New CoreContainer 1956333402
   [junit4]   2> 58273 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/]
   [junit4]   2> 58274 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 58275 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 58275 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 58275 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 58275 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 58276 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 58276 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 58276 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 58277 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 58280 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 58281 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 58281 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 58281 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49340/solr
   [junit4]   2> 58282 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 58283 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 58285 T135 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e69a184 name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 58286 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 58289 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 58291 T137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c710710 name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 58292 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 58299 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 58336 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58337 T137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58336 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58336 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58336 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58353 T124 oasc.ZkController.register We are http://127.0.0.1:49022/sm_gby/a/collection1/ and leader is http://127.0.0.1:49022/sm_gby/a/collection1/
   [junit4]   2> 58354 T124 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49022/sm_gby/a
   [junit4]   2> 58354 T124 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 58354 T124 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 58354 T124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 58357 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58358 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58358 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58359 T124 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 58361 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 58363 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49022/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49022_sm_gby%2Fa",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 58366 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58472 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58472 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58472 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58472 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58472 T137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 59306 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52027_sm_gby%2Fa
   [junit4]   2> 59307 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52027_sm_gby%2Fa
   [junit4]   2> 59312 T101 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59313 T79 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59312 T88 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59313 T137 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59313 T119 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59324 T138 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 59324 T138 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 59326 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59327 T138 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 59327 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59327 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59329 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 59331 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52027/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52027_sm_gby%2Fa",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 59331 T80 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 59331 T80 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 59335 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59438 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 59438 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 59439 T137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 59439 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 59439 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 60327 T138 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 60328 T138 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/collection1
   [junit4]   2> 60328 T138 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 60329 T138 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 60330 T138 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 60331 T138 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 60332 T138 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/collection1/'
   [junit4]   2> 60379 T138 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 60419 T138 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 60421 T138 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 60448 T138 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 60801 T138 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 60804 T138 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 60806 T138 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 60806 T138 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 60807 T138 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 60807 T138 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/
   [junit4]   2> 60808 T138 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 60809 T138 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3
   [junit4]   2> 60809 T138 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index/
   [junit4]   2> 60809 T138 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 60809 T138 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index
   [junit4]   2> 60811 T138 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6d66cc49 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 60811 T138 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 60812 T138 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 60813 T138 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 60813 T138 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 60813 T138 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 60813 T138 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 60814 T138 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 60814 T138 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 60814 T138 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 60817 T138 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 60819 T138 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 60820 T138 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 60820 T138 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6d66cc49 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 60821 T138 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 60821 T138 oass.SolrIndexSearcher.<init> Opening Searcher@55c9e26f[collection1] main
   [junit4]   2> 60822 T139 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@55c9e26f[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 60822 T138 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 60824 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 60824 T142 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52027/sm_gby/a collection:collection1 shard:shard2
   [junit4]   2> 60824 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 60835 T142 oasc.ZkController.register We are http://127.0.0.1:52027/sm_gby/a/collection1/ and leader is http://127.0.0.1:54864/sm_gby/a/collection1/
   [junit4]   2> 60835 T142 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52027/sm_gby/a
   [junit4]   2> 60835 T142 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 60836 T142 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@48ccb726 url=http://127.0.0.1:52027/sm_gby/a/collection1 node=127.0.0.1:52027_sm_gby%2Fa C9_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52027/sm_gby/a, core=collection1, node_name=127.0.0.1:52027_sm_gby%2Fa}
   [junit4]   2> 60841 T143 C9 P52027 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 60843 T143 C9 P52027 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 60843 T142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 60844 T143 C9 P52027 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 60844 T143 C9 P52027 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 60846 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 60846 T79 oasc.DistributedQueue$LatchChildWa

[...truncated too long message...]

tate=active, base_url=http://127.0.0.1:52027/sm_gby/a, core=collection1, node_name=127.0.0.1:52027_sm_gby%2Fa}
   [junit4]   2> 72790 T137 C21 P52027 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 71867 T119 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 71778 T101 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80775 T101 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80781 T156 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 80781 T156 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6c9d1e9b
   [junit4]   2> 80781 T137 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 80782 T137 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@48ccb726
   [junit4]   2> 80783 T156 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 80784 T50 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 80785 T137 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 80784 T156 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 80786 T156 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 80786 T156 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 80785 T137 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 80787 T137 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 80788 T137 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 80789 T156 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 80790 T156 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 80790 T156 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4;done=false>>]
   [junit4]   2> 80791 T137 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 80791 T156 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4
   [junit4]   2> 80791 T137 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 80792 T156 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4/index;done=false>>]
   [junit4]   2> 80792 T156 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4/index
   [junit4]   2> 80792 T137 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index;done=false>>]
   [junit4]   2> 80794 T137 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index
   [junit4]   2> 80793 T156 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80794 T137 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3;done=false>>]
   [junit4]   2> 80795 T156 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 80795 T137 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3
   [junit4]   2> 80797 T137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80797 T137 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80798 T137 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 80799 T137 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=vi_VN, timezone=Pacific/Marquesas
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=92268280,total=126484480
   [junit4]   2> NOTE: All tests run in this JVM: [LargeVolumeJettyTest, LargeVolumeEmbeddedTest, TestDocumentObjectBinder, CloudSolrServerTest]
   [junit4] Completed on J0 in 71.42s, 2 tests, 1 error <<< FAILURES!

[...truncated 109 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:451: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1278: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:911: There were test failures: 48 suites, 280 tests, 1 error

Total time: 48 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:+UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure