You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/18 05:09:00 UTC

[JENKINS] Lucene-Solr-4.x-Linux (64bit/ibm-j9-jdk7) - Build # 9405 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9405/
Java: 64bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestCoreContainer

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([ADDB06CE0FC8C9B3]:0)


REGRESSION:  org.apache.solr.core.TestCoreContainer.testReloadThreaded

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([ADDB06CE0FC8C9B3]:0)


REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection

Error Message:
expected:<1> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<3>
	at __randomizedtesting.SeedInfo.seed([ADDB06CE0FC8C9B3:F6EBFD060C8F6ACF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([ADDB06CE0FC8C9B3:CC4EDC2EE719C345]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded(SignatureUpdateProcessorFactoryTest.java:222)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)




Build Log:
[...truncated 10532 lines...]
   [junit4] Suite: org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest
   [junit4]   2> 853563 T2613 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 853564 T2613 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727
   [junit4]   2> 853564 T2613 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 853565 T2613 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 853565 T2613 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 853599 T2613 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 853646 T2613 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 853647 T2613 oass.IndexSchema.readSchema Reading Solr Schema from schema12.xml
   [junit4]   2> 853652 T2613 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 853988 T2613 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 853991 T2613 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 853992 T2613 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 853997 T2613 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 854000 T2613 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 854003 T2613 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 854004 T2613 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 854004 T2613 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 854042 T2613 oasc.CoreContainer.<init> New CoreContainer -1101667118
   [junit4]   2> 854042 T2613 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 854043 T2613 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 854044 T2613 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 854044 T2613 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 854044 T2613 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 854044 T2613 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 854045 T2613 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 854045 T2613 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 854045 T2613 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 854046 T2613 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 854050 T2613 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 854050 T2613 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 854050 T2613 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 854053 T2614 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 854053 T2614 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 854054 T2614 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 854054 T2614 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 854094 T2614 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 854143 T2614 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 854143 T2614 oass.IndexSchema.readSchema Reading Solr Schema from schema12.xml
   [junit4]   2> 854149 T2614 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 854453 T2614 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 854455 T2614 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 854456 T2614 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 854461 T2614 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 854464 T2614 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 854467 T2614 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 854467 T2614 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727/
   [junit4]   2> 854467 T2614 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@240069d5
   [junit4]   2> 854468 T2614 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727
   [junit4]   2> 854468 T2614 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727/index/
   [junit4]   2> 854468 T2614 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727/index' doesn't exist. Creating new index...
   [junit4]   2> 854469 T2614 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727/index
   [junit4]   2> 854469 T2614 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=83.1220703125, floorSegmentMB=1.162109375, forceMergeDeletesPctAllowed=17.092337228227635, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 854470 T2614 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_1,generation=1}
   [junit4]   2> 854470 T2614 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 854470 T2614 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 854472 T2614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 854472 T2614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 854472 T2614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 854473 T2614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 854473 T2614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 854473 T2614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 854473 T2614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 854474 T2614 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 854474 T2614 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 854474 T2614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 854474 T2614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 854475 T2614 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 854475 T2614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 854475 T2614 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 854475 T2614 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 854476 T2614 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 854476 T2614 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 854476 T2614 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 854477 T2614 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 854477 T2614 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 854477 T2614 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 854478 T2614 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 854478 T2614 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 854478 T2614 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 854479 T2614 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 854479 T2614 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 854479 T2614 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 854479 T2614 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 854480 T2614 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 854480 T2614 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 854480 T2614 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 854481 T2614 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 854481 T2614 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 854481 T2614 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 854482 T2614 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 854482 T2614 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 854483 T2614 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 854484 T2614 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 854486 T2614 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 854486 T2614 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 854486 T2614 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 854487 T2614 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_1,generation=1}
   [junit4]   2> 854487 T2614 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 854487 T2614 oass.SolrIndexSearcher.<init> Opening Searcher@9bbdc80d[collection1] main
   [junit4]   2> 854488 T2614 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 854538 T2614 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 854559 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9bbdc80d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 854562 T2614 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 854564 T2613 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 854569 T2613 oas.SolrTestCaseJ4.setUp ###Starting testDupeDetection
   [junit4]   2> ASYNC  NEW_CORE C1370 name=collection1 org.apache.solr.core.SolrCore@fa4a81cb
   [junit4]   2> 854570 T2613 C1370 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 854570 T2613 C1370 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 0
   [junit4]   2> 854573 T2613 C1370 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854574 T2613 C1370 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_2,generation=2}
   [junit4]   2> 854574 T2613 C1370 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 854575 T2613 C1370 oass.SolrIndexSearcher.<init> Opening Searcher@8eece05a[collection1] main
   [junit4]   2> 854576 T2613 C1370 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854577 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8eece05a[collection1] main{StandardDirectoryReader(segments_2:2:nrt)}
   [junit4]   2> 854577 T2613 C1370 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 854581 T2613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854583 T2613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_3,generation=3}
   [junit4]   2> 854583 T2613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 854584 T2613 oass.SolrIndexSearcher.<init> Opening Searcher@a955bc0d[collection1] main
   [junit4]   2> 854584 T2613 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854586 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a955bc0d[collection1] main{StandardDirectoryReader(segments_3:4:nrt _0(4.7):c2)}
   [junit4]   2> 854589 T2613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854590 T2613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_3,generation=3}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_4,generation=4}
   [junit4]   2> 854591 T2613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
   [junit4]   2> 854591 T2613 oass.SolrIndexSearcher.<init> Opening Searcher@f937a4c[collection1] main
   [junit4]   2> 854592 T2613 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854594 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f937a4c[collection1] main{StandardDirectoryReader(segments_4:6:nrt _0(4.7):c2 _1(4.7):c1)}
   [junit4]   2> 854594 T2613 oas.SolrTestCaseJ4.tearDown ###Ending testDupeDetection
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SignatureUpdateProcessorFactoryTest -Dtests.method=testDupeDetection -Dtests.seed=ADDB06CE0FC8C9B3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de_DE -Dtests.timezone=US/Central -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.03s J1 | SignatureUpdateProcessorFactoryTest.testDupeDetection <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1> but was:<3>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ADDB06CE0FC8C9B3:F6EBFD060C8F6ACF]:0)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 854601 T2613 oas.SolrTestCaseJ4.setUp ###Starting testFailNonIndexedSigWithOverwriteDupes
   [junit4]   2> 854601 T2613 C1370 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 854602 T2613 C1370 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 854603 T2613 C1370 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854604 T2613 C1370 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_4,generation=4}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_5,generation=5}
   [junit4]   2> 854604 T2613 C1370 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
   [junit4]   2> 854605 T2613 C1370 oass.SolrIndexSearcher.<init> Opening Searcher@48ad5dd7[collection1] main
   [junit4]   2> 854605 T2613 C1370 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854606 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@48ad5dd7[collection1] main{StandardDirectoryReader(segments_5:7:nrt)}
   [junit4]   2> 854607 T2613 C1370 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 854608 T2613 oas.SolrTestCaseJ4.tearDown ###Ending testFailNonIndexedSigWithOverwriteDupes
   [junit4]   2> 854611 T2613 oas.SolrTestCaseJ4.setUp ###Starting testMultiThreaded
   [junit4]   2> 854611 T2613 C1370 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 854612 T2613 C1370 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 854613 T2613 C1370 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854613 T2613 C1370 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_5,generation=5}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_6,generation=6}
   [junit4]   2> 854614 T2613 C1370 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
   [junit4]   2> 854614 T2613 C1370 oass.SolrIndexSearcher.<init> Opening Searcher@9ec253e5[collection1] main
   [junit4]   2> 854615 T2613 C1370 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854616 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9ec253e5[collection1] main{StandardDirectoryReader(segments_6:8:nrt)}
   [junit4]   2> 854617 T2613 C1370 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 854627 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854633 T2625 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_6,generation=6}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_7,generation=7}
   [junit4]   2> 854633 T2625 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
   [junit4]   2> 854639 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@51c0606a[collection1] main
   [junit4]   2> 854639 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854640 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854640 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51c0606a[collection1] main{StandardDirectoryReader(segments_7:29:nrt _8(4.7):c11/9:delGen=1)}
   [junit4]   2> 854646 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_7,generation=7}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_8,generation=8}
   [junit4]   2> 854647 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 8
   [junit4]   2> 854658 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@2202ee7a[collection1] main
   [junit4]   2> 854658 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854659 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854660 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2202ee7a[collection1] main{StandardDirectoryReader(segments_8:48:nrt _c(4.7):c2/1:delGen=1)}
   [junit4]   2> 854668 T2626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_8,generation=8}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_9,generation=9}
   [junit4]   2> 854669 T2626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 9
   [junit4]   2> 854679 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@df8afa63[collection1] main
   [junit4]   2> 854684 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854685 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854686 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@df8afa63[collection1] main{StandardDirectoryReader(segments_9:65:nrt _k(4.7):c14/13:delGen=1 _m(4.7):c4/3:delGen=1)}
   [junit4]   2> 854695 T2625 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_9,generation=9}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_a,generation=10}
   [junit4]   2> 854696 T2625 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 10
   [junit4]   2> 854706 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@cf43d68d[collection1] main
   [junit4]   2> 854707 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854708 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854709 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@cf43d68d[collection1] main{StandardDirectoryReader(segments_a:81:nrt _r(4.7):c16/15:delGen=1 _q(4.7):c5/4:delGen=1)}
   [junit4]   2> 854717 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_a,generation=10}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_b,generation=11}
   [junit4]   2> 854718 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 11
   [junit4]   2> 854723 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@f2401db[collection1] main
   [junit4]   2> 854724 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854724 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854725 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f2401db[collection1] main{StandardDirectoryReader(segments_b:97:nrt _w(4.7):c8/6:delGen=1)}
   [junit4]   2> 854726 T2626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_b,generation=11}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_c,generation=12}
   [junit4]   2> 854726 T2626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 12
   [junit4]   2> 854727 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@889a0faa[collection1] main
   [junit4]   2> 854727 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854727 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854728 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@889a0faa[collection1] main{StandardDirectoryReader(segments_c:99:nrt _w(4.7):c8/7:delGen=2 _10(4.7):c1)}
   [junit4]   2> 854729 T2625 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_c,generation=12}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_d,generation=13}
   [junit4]   2> 854729 T2625 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 13
   [junit4]   2> 854730 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@24929ebc[collection1] main
   [junit4]   2> 854730 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854731 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854731 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24929ebc[collection1] main{StandardDirectoryReader(segments_d:103:nrt _w(4.7):c8/7:delGen=2 _11(4.7):c1)}
   [junit4]   2> 854732 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_d,generation=13}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_e,generation=14}
   [junit4]   2> 854733 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 14
   [junit4]   2> 854733 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@e6ab3ed9[collection1] main
   [junit4]   2> 854734 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854734 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854735 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e6ab3ed9[collection1] main{StandardDirectoryReader(segments_e:107:nrt _w(4.7):c8/7:delGen=2 _12(4.7):c1)}
   [junit4]   2> 854735 T2626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_e,generation=14}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_f,generation=15}
   [junit4]   2> 854736 T2626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 15
   [junit4]   2> 854736 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@4cf6132a[collection1] main
   [junit4]   2> 854737 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854737 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854738 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4cf6132a[collection1] main{StandardDirectoryReader(segments_f:111:nrt _12(4.7):c1 _13(4.7):c1)}
   [junit4]   2> 854739 T2625 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_f,generation=15}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_g,generation=16}
   [junit4]   2> 854739 T2625 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 16
   [junit4]   2> 854740 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@d8c03518[collection1] main
   [junit4]   2> 854741 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854741 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854742 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d8c03518[collection1] main{StandardDirectoryReader(segments_g:117:nrt _14(4.7):c1)}
   [junit4]   2> 854743 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_g,generation=16}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_h,generation=17}
   [junit4]   2> 854743 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 17
   [junit4]   2> 854745 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@12476dc7[collection1] main
   [junit4]   2> 854745 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854745 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854746 T2626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_h,generation=17}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_i,generation=18}
   [junit4]   2> 854746 T2626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 18
   [junit4]   2> 854746 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12476dc7[collection1] main{StandardDirectoryReader(segments_h:125:nrt _16(4.7):c1)}
   [junit4]   2> 854747 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@771bd02f[collection1] main
   [junit4]   2> 854747 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854748 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854748 T2625 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 854749 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@771bd02f[collection1] main{StandardDirectoryReader(segments_h:125:nrt _16(4.7):c1)}
   [junit4]   2> 854750 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@104b4858[collection1] main
   [junit4]   2> 854751 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854751 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854752 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@104b4858[collection1] main{StandardDirectoryReader(segments_i:130:nrt _17(4.7):c1)}
   [junit4]   2> 854753 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_i,generation=18}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_j,generation=19}
   [junit4]   2> 854753 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 19
   [junit4]   2> 854753 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@9b2dba87[collection1] main
   [junit4]   2> 854754 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854754 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854755 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9b2dba87[collection1] main{StandardDirectoryReader(segments_j:133:nrt _18(4.7):c1)}
   [junit4]   2> 854756 T2626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_j,generation=19}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_k,generation=20}
   [junit4]   2> 854756 T2626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 20
   [junit4]   2> 854756 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@6f0b4535[collection1] main
   [junit4]   2> 854757 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854757 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854758 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6f0b4535[collection1] main{StandardDirectoryReader(segments_k:137:nrt _19(4.7):c1)}
   [junit4]   2> 854759 T2625 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_k,generation=20}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_l,generation=21}
   [junit4]   2> 854759 T2625 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 21
   [junit4]   2> 854760 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@26648f1f[collection1] main
   [junit4]   2> 854761 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854761 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854762 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@26648f1f[collection1] main{StandardDirectoryReader(segments_l:141:nrt _1a(4.7):c1)}
   [junit4]   2> 854763 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_l,generation=21}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_m,generation=22}
   [junit4]   2> 854763 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 22
   [junit4]   2> 854765 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@f601290e[collection1] main
   [junit4]   2> 854765 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854765 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854766 T2626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_m,generation=22}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_n,generation=23}
   [junit4]   2> 854766 T2626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 23
   [junit4]   2> 854766 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f601290e[collection1] main{StandardDirectoryReader(segments_m:149:nrt _1c(4.7):c1)}
   [junit4]   2> 854767 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@eea45a25[collection1] main
   [junit4]   2> 854767 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854768 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854768 T2625 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 854769 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@eea45a25[collection1] main{StandardDirectoryReader(segments_m:149:nrt _1c(4.7):c1)}
   [junit4]   2> 854770 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@201be310[collection1] main
   [junit4]   2> 854771 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854771 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854772 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@201be310[collection1] main{StandardDirectoryReader(segments_n:154:nrt _1d(4.7):c1)}
   [junit4]   2> 854773 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_n,generation=23}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_o,generation=24}
   [junit4]   2> 854773 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 24
   [junit4]   2> 854773 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@c653be02[collection1] main
   [junit4]   2> 854774 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854774 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854775 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c653be02[collection1] main{StandardDirectoryReader(segments_o:157:nrt _1e(4.7):c1)}
   [junit4]   2> 854775 T2626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_o,generation=24}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_p,generation=25}
   [junit4]   2> 854776 T2626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 25
   [junit4]   2> 854776 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@7ad29630[collection1] main
   [junit4]   2> 854777 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854777 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854778 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7ad29630[collection1] main{StandardDirectoryReader(segments_p:159:nrt _1e(4.7):c1 _1f(4.7):c1)}
   [junit4]   2> 854779 T2625 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_p,generation=25}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_q,generation=26}
   [junit4]   2> 854779 T2625 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 26
   [junit4]   2> 854780 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@bab1a438[collection1] main
   [junit4]   2> 854781 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854781 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854782 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bab1a438[collection1] main{StandardDirectoryReader(segments_q:166:nrt _1g(4.7):c1)}
   [junit4]   2> 854783 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_q,generation=26}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_r,generation=27}
   [junit4]   2> 854783 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 27
   [junit4]   2> 854783 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@29c01b4e[collection1] main
   [junit4]   2> 854784 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854784 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854785 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29c01b4e[collection1] main{StandardDirectoryReader(segments_r:169:nrt _1h(4.7):c1)}
   [junit4]   2> 854786 T2626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_r,generation=27}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_s,generation=28}
   [junit4]   2> 854786 T2626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 28
   [junit4]   2> 854786 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@9c9c63a[collection1] main
   [junit4]   2> 854787 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854787 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854788 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9c9c63a[collection1] main{StandardDirectoryReader(segments_s:171:nrt _1h(4.7):c1 _1i(4.7):c1)}
   [junit4]   2> 854789 T2625 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_s,generation=28}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_t,generation=29}
   [junit4]   2> 854789 T2625 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 29
   [junit4]   2> 854790 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@b0a4ec93[collection1] main
   [junit4]   2> 854791 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854791 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854792 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b0a4ec93[collection1] main{StandardDirectoryReader(segments_t:175:nrt _1h(4.7):c1 _1j(4.7):c1)}
   [junit4]   2> 854793 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_t,generation=29}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_u,generation=30}
   [junit4]   2> 854794 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 30
   [junit4]   2> 854795 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@643a89d6[collection1] main
   [junit4]   2> 854796 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854796 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854796 T2626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_u,generation=30}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_v,generation=31}
   [junit4]   2> 854797 T2626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 31
   [junit4]   2> 854797 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@643a89d6[collection1] main{StandardDirectoryReader(segments_u:183:nrt _1k(4.7):c1 _1l(4.7):c1)}
   [junit4]   2> 854797 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@fb118565[collection1] main
   [junit4]   2> 854798 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854798 T2625 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854799 T2625 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 854799 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fb118565[collection1] main{StandardDirectoryReader(segments_u:183:nrt _1k(4.7):c1 _1l(4.7):c1)}
   [junit4]   2> 854801 T2625 oass.SolrIndexSearcher.<init> Opening Searcher@707d177[collection1] main
   [junit4]   2> 854801 T2625 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854802 T2624 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854802 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@707d177[collection1] main{StandardDirectoryReader(segments_v:188:nrt _1k(4.7):c1 _1m(4.7):c1)}
   [junit4]   2> 854803 T2624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_v,generation=31}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_w,generation=32}
   [junit4]   2> 854804 T2624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 32
   [junit4]   2> 854804 T2624 oass.SolrIndexSearcher.<init> Opening Searcher@db4c466e[collection1] main
   [junit4]   2> 854805 T2624 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854805 T2626 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854806 T2626 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 854806 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@db4c466e[collection1] main{StandardDirectoryReader(segments_w:191:nrt _1k(4.7):c1 _1n(4.7):c1)}
   [junit4]   2> 854806 T2626 oass.SolrIndexSearcher.<init> Opening Searcher@bbda5ad7[collection1] main
   [junit4]   2> 854807 T2626 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854808 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bbda5ad7[collection1] main{StandardDirectoryReader(segments_w:191:nrt _1k(4.7):c1 _1n(4.7):c1)}
   [junit4]   2> ASYNC  NEW_CORE C1371 name=collection1 org.apache.solr.core.SolrCore@fa4a81cb
   [junit4]   2> 854808 T2613 C1371 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854809 T2613 C1371 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 854809 T2613 C1371 oass.SolrIndexSearcher.<init> Opening Searcher@71d4ecb9[collection1] main
   [junit4]   2> 854810 T2613 C1371 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854811 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@71d4ecb9[collection1] main{StandardDirectoryReader(segments_w:191:nrt _1k(4.7):c1 _1n(4.7):c1)}
   [junit4]   2> 854811 T2613 C1371 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 3
   [junit4]   2> 854813 T2613 oas.SolrTestCaseJ4.tearDown ###Ending testMultiThreaded
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SignatureUpdateProcessorFactoryTest -Dtests.method=testMultiThreaded -Dtests.seed=ADDB06CE0FC8C9B3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de_DE -Dtests.timezone=US/Central -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.21s J1 | SignatureUpdateProcessorFactoryTest.testMultiThreaded <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1> but was:<2>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ADDB06CE0FC8C9B3:CC4EDC2EE719C345]:0)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded(SignatureUpdateProcessorFactoryTest.java:222)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 854818 T2613 oas.SolrTestCaseJ4.setUp ###Starting testNonIndexedSignatureField
   [junit4]   2> 854818 T2613 C1371 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 854819 T2613 C1371 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 854820 T2613 C1371 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854821 T2613 C1371 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_w,generation=32}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_x,generation=33}
   [junit4]   2> 854821 T2613 C1371 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 33
   [junit4]   2> 854822 T2613 C1371 oass.SolrIndexSearcher.<init> Opening Searcher@24e6c3c7[collection1] main
   [junit4]   2> 854822 T2613 C1371 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854823 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24e6c3c7[collection1] main{StandardDirectoryReader(segments_x:192:nrt)}
   [junit4]   2> 854824 T2613 C1371 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 4
   [junit4]   2> 854828 T2613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854829 T2613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_x,generation=33}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_y,generation=34}
   [junit4]   2> 854830 T2613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 34
   [junit4]   2> 854831 T2613 oass.SolrIndexSearcher.<init> Opening Searcher@5cae9033[collection1] main
   [junit4]   2> 854831 T2613 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854832 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5cae9033[collection1] main{StandardDirectoryReader(segments_y:194:nrt _1o(4.7):c2)}
   [junit4]   2> 854833 T2613 oas.SolrTestCaseJ4.tearDown ###Ending testNonIndexedSignatureField
   [junit4]   2> 854836 T2613 oas.SolrTestCaseJ4.setUp ###Starting testNonStringFieldsValues
   [junit4]   2> 854837 T2613 C1371 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 854837 T2613 C1371 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 0
   [junit4]   2> 854839 T2613 C1371 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 854839 T2613 C1371 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_y,generation=34}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_z,generation=35}
   [junit4]   2> 854840 T2613 C1371 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 35
   [junit4]   2> 854840 T2613 C1371 oass.SolrIndexSearcher.<init> Opening Searcher@a0a83cd6[collection1] main
   [junit4]   2> 854841 T2613 C1371 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 854842 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a0a83cd6[collection1] main{StandardDirectoryReader(segments_z:195:nrt)}
   [junit4]   2> 854842 T2613 C1371 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 3
   [junit4]   2> 854847 T2613 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 855376 T2613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 855378 T2613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_z,generation=35}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_10,generation=36}
   [junit4]   2> 855379 T2613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 36
   [junit4]   2> 855380 T2613 oass.SolrIndexSearcher.<init> Opening Searcher@2587bbe5[collection1] main
   [junit4]   2> 855380 T2613 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 855382 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2587bbe5[collection1] main{StandardDirectoryReader(segments_10:197:nrt _1p(4.7):c5/1:delGen=1)}
   [junit4]   2> 855383 T2613 oas.SolrTestCaseJ4.tearDown ###Ending testNonStringFieldsValues
   [junit4]   2> 855386 T2613 oas.SolrTestCaseJ4.setUp ###Starting testDupeAllFieldsDetection
   [junit4]   2> 855387 T2613 C1371 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 855387 T2613 C1371 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 0
   [junit4]   2> 855389 T2613 C1371 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 855389 T2613 C1371 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_10,generation=36}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_11,generation=37}
   [junit4]   2> 855390 T2613 C1371 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 37
   [junit4]   2> 855391 T2613 C1371 oass.SolrIndexSearcher.<init> Opening Searcher@9ffcbb5b[collection1] main
   [junit4]   2> 855391 T2613 C1371 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 855393 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9ffcbb5b[collection1] main{StandardDirectoryReader(segments_11:198:nrt)}
   [junit4]   2> 855394 T2613 C1371 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 5
   [junit4]   2> 855399 T2613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 855400 T2613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_11,generation=37}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@d2fb667f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8a0596bc)),segFN=segments_12,generation=38}
   [junit4]   2> 855400 T2613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 38
   [junit4]   2> 855401 T2613 oass.SolrIndexSearcher.<init> Opening Searcher@f4161684[collection1] main
   [junit4]   2> 855402 T2613 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 855404 T2615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f4161684[collection1] main{StandardDirectoryReader(segments_12:200:nrt _1q(4.7):c3)}
   [junit4]   2> 855404 T2613 oas.SolrTestCaseJ4.tearDown ###Ending testDupeAllFieldsDetection
   [junit4]   2> 855406 T2613 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 855406 T2613 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1101667118
   [junit4]   2> 855406 T2613 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@fa4a81cb
   [junit4]   2> 855414 T2613 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=42,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=253,cumulative_deletesById=0,cumulative_deletesByQuery=6,cumulative_errors=0}
   [junit4]   2> 855415 T2613 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 855415 T2613 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 855415 T2613 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 855416 T2613 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 855416 T2613 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 855416 T2613 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727/index;done=false>>]
   [junit4]   2> 855417 T2613 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727/index
   [junit4]   2> 855417 T2613 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727;done=false>>]
   [junit4]   2> 855417 T2613 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1392689846727
   [junit4]   2> NOTE: test params are: codec=Lucene46: {signatureField=PostingsFormat(name=Asserting), v_t=FST41, weight=Lucene41(blocksize=128), ints_is=PostingsFormat(name=MockRandom), id=FST41, name=FST41}, docValues:{}, sim=DefaultSimilarity, locale=de_DE, timezone=US/Central
   [junit4]   2> NOTE: Linux 3.8.0-35-generic amd64/IBM Corporation 1.7.0 (64-bit)/cpus=8,threads=1,free=36537912,total=263913472
   [junit4]   2> NOTE: All tests run in this JVM: [TestPartialUpdateDeduplication, SuggesterTSTTest, TestReload, ZkNodePropsTest, TestFastOutputStream, TestStressReorder, TestBinaryResponseWriter, SuggestComponentTest, IndexBasedSpellCheckerTest, TestFastWriter, TestFreeTextSuggestions, TestCloudManagedSchema, TestFileDictionaryLookup, TestZkChroot, TestRTGBase, NotRequiredUniqueKeyTest, TestComponentsName, TestSolrJ, TestHashPartitioner, TestXIncludeConfig, TestBlendedInfixSuggestions, CoreAdminCreateDiscoverTest, TestDefaultSimilarityFactory, TestDynamicFieldResource, XsltUpdateRequestHandlerTest, TestDistributedGrouping, TestHighlightDedupGrouping, TestCSVLoader, UniqFieldsUpdateProcessorFactoryTest, TestPhraseSuggestions, CoreContainerCoreInitFailuresTest, TestFieldCollectionResource, AddSchemaFieldsUpdateProcessorFactoryTest, TestFieldTypeResource, TestSolrDeletionPolicy2, LeaderElectionIntegrationTest, PingRequestHandlerTest, DisMaxRequestHandlerTest, SolrCoreCheckLockOnStartupTest, TestManagedSchema, InfoHandlerTest, PluginInfoTest, ZkSolrClientTest, SolrCmdDistributorTest, AlternateDirectoryTest, TestCSVResponseWriter, StatsComponentTest, TestCollationFieldDocValues, XmlUpdateRequestHandlerTest, ParsingFieldUpdateProcessorsTest, TestManagedSchemaFieldResource, SpellCheckCollatorTest, TestIBSimilarityFactory, TestStressUserVersions, SpellingQueryConverterTest, RemoteQueryErrorTest, TestFoldingMultitermQuery, TestNoOpRegenerator, ShardRoutingCustomTest, TestCollationField, OpenExchangeRatesOrgProviderTest, SolrIndexConfigTest, CurrencyFieldOpenExchangeTest, TestArbitraryIndexDir, TestNumberUtils, ChaosMonkeyNothingIsSafeTest, TestDFRSimilarityFactory, TestDistributedSearch, OpenCloseCoreStressTest, TestFastLRUCache, TestFuzzyAnalyzedSuggestions, TestMultiCoreConfBootstrap, TermVectorComponentDistributedTest, TestDistributedMissingSort, TestQueryTypes, TestWordDelimiterFilterFactory, DeleteReplicaTest, TestHighFrequencyDictionaryFactory, TestStandardQParsers, DocValuesMultiTest, DebugComponentTest, TestAnalyzedSuggestions, BadIndexSchemaTest, TestOmitPositions, DefaultValueUpdateProcessorTest, TestStressRecovery, SolrRequestParserTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, BasicZkTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, TestRandomFaceting, LeaderElectionTest, TestRandomDVFaceting, AutoCommitTest, TestFaceting, TestRecovery, DistributedSpellCheckComponentTest, TestJoin, HardAutoCommitTest, TestRangeQuery, SimpleFacetsTest, TestSolr4Spatial, SolrCoreTest, SpellCheckComponentTest, QueryElevationComponentTest, ConvertedLegacyTest, TestSort, TestFiltering, BasicFunctionalityTest, TestLazyCores, DirectUpdateHandlerTest, TestBadConfig, TestIndexSearcher, HighlighterTest, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest]
   [junit4] Completed on J1 in 1.87s, 6 tests, 2 failures <<< FAILURES!

[...truncated 530 lines...]
   [junit4] Suite: org.apache.solr.core.TestCoreContainer
   [junit4]   2> 103643 T417 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 103644 T417 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-TestCoreContainer-1392689096805
   [junit4]   2> 103644 T417 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 103645 T417 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 103646 T417 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 103683 T417 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 103730 T417 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 103731 T417 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 103738 T417 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 104085 T417 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 104086 T417 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 104086 T417 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 104093 T417 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 104096 T417 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 104104 T417 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 104110 T417 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 104115 T417 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 104116 T417 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 104116 T417 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 104117 T417 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 104117 T417 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 104117 T417 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 104118 T417 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 104118 T417 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 104119 T417 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 104159 T417 oasc.CoreContainer.<init> New CoreContainer -1500146813
   [junit4]   2> 104159 T417 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 104160 T417 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 104161 T417 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 104161 T417 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 104161 T417 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 104162 T417 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 104163 T417 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 104163 T417 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 104164 T417 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 104164 T417 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 104170 T417 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 104170 T417 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 104170 T417 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 104174 T418 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 104174 T418 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 104175 T418 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to c

[...truncated too long message...]

olrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 7323730 T417 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 7323730 T417 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 7323731 T417 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.core.TestCoreContainer_reloadThreaded/collection1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.core.TestCoreContainer_reloadThreaded/collection1/data/index;done=false>>]
   [junit4]   2> 7323731 T417 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.core.TestCoreContainer_reloadThreaded/collection1/data/index
   [junit4]   2> 7323732 T417 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.core.TestCoreContainer_reloadThreaded/collection1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.core.TestCoreContainer_reloadThreaded/collection1/data;done=false>>]
   [junit4]   2> 7323732 T417 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.core.TestCoreContainer_reloadThreaded/collection1/data
   [junit4]   2> 7323733 T417 oas.SolrTestCaseJ4.tearDown ###Ending testReloadThreaded
   [junit4]   2> 7323736 T417 oas.SolrTestCaseJ4.setUp ###Starting testReloadSequential
   [junit4]   2> 7323739 T417 oas.SolrTestCaseJ4.tearDown ###Ending testReloadSequential
   [junit4]   2> 7323744 T417 oas.SolrTestCaseJ4.setUp ###Starting testCustomHandlers
   [junit4]   2> 7323745 T417 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/collection1/'
   [junit4]   2> 7323796 T417 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr/collection1
   [junit4]   2> 7323796 T417 oasc.CoreContainer.<init> New CoreContainer -715114234
   [junit4]   2> 7323797 T417 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=solr/collection1/]
   [junit4]   2> 7323798 T417 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 7323798 T417 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 7323799 T417 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 7323799 T417 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 7323799 T417 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 7323800 T417 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 7323800 T417 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 7323800 T417 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 7323801 T417 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 7323807 T417 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 7323808 T417 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 7323808 T417 oasc.CoreContainer.load Host Name: null
   [junit4]   2> 7323811 T417 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr/collection1
   [junit4]   2> 7323812 T417 oasc.CorePropertiesLocator.discover Found 0 core definitions
   [junit4]   2> 7323813 T417 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-715114234
   [junit4]   2> 7323813 T417 oas.SolrTestCaseJ4.tearDown ###Ending testCustomHandlers
   [junit4]   2> 7323815 T417 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 7323815 T417 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1500146813
   [junit4]   2> 7323815 T417 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2cb4f645
   [junit4]   2> 7323816 T417 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 7323817 T417 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 7323817 T417 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 7323817 T417 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 7323818 T417 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 7323818 T417 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 7323819 T417 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-TestCoreContainer-1392689096805/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-TestCoreContainer-1392689096805/index;done=false>>]
   [junit4]   2> 7323819 T417 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-TestCoreContainer-1392689096805/index
   [junit4]   2> 7323820 T417 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-TestCoreContainer-1392689096805 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-TestCoreContainer-1392689096805;done=false>>]
   [junit4]   2> 7323820 T417 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-TestCoreContainer-1392689096805
   [junit4]   2> 7220185 T416 ccr.ThreadLeakControl.tryToInterruptAll All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=en_HK, timezone=CTT
   [junit4]   2> NOTE: Linux 3.8.0-35-generic amd64/IBM Corporation 1.7.0 (64-bit)/cpus=8,threads=1,free=34294856,total=97976320
   [junit4]   2> NOTE: All tests run in this JVM: [FieldMutatingUpdateProcessorTest, TestLMDirichletSimilarityFactory, QueryParsingTest, SliceStateUpdateTest, SystemInfoHandlerTest, CircularListTest, TestExtendedDismaxParser, DistributedSuggestComponentTest, CoreMergeIndexesAdminHandlerTest, TestValueSourceCache, DOMUtilTest, SpellPossibilityIteratorTest, TestSolrQueryParserDefaultOperatorResource, FileBasedSpellCheckerTest, SliceStateTest, TestCloudManagedSchemaAddField, TestDynamicFieldCollectionResource, DocValuesTest, TestAddFieldRealTimeGet, ShardRoutingTest, TestJmxMonitoredMap, CacheHeaderTest, BinaryUpdateRequestHandlerTest, TestQueryUtils, TermVectorComponentTest, TestSolrXml, SolrTestCaseJ4Test, TestGroupingSearch, TestSolrIndexConfig, RegexBoostProcessorTest, SuggesterFSTTest, ZkCLITest, DistributedDebugComponentTest, TermsComponentTest, DocumentBuilderTest, FastVectorHighlighterTest, TestCoreContainer]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCoreContainer -Dtests.seed=ADDB06CE0FC8C9B3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_HK -Dtests.timezone=CTT -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestCoreContainer (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ADDB06CE0FC8C9B3]:0)
   [junit4] Completed on J0 in 7220.21s, 6 tests, 2 errors <<< FAILURES!

[...truncated 175 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:452: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 362 suites, 1561 tests, 1 suite-level error, 1 error, 2 failures, 71 ignored (7 assumptions)

Total time: 149 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60-ea-b04) - Build # 9410 - Still Failing!

Posted by Simon Willnauer <si...@gmail.com>.
I committed a fix for this - sorry for the noise here!

On Tue, Feb 18, 2014 at 6:04 PM, Mark Miller <ma...@gmail.com> wrote:
> Fail is:
>
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8
>
> - Mark
>
> http://about.me/markrmiller
>
> On Feb 18, 2014, at 12:00 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:
>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9410/
>> Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
>>
>> All tests passed
>>
>> Build Log:
>> [...truncated 28008 lines...]
>> BUILD FAILED
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:64: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8
>>
>> Total time: 58 minutes 12 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Description set: Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60-ea-b04) - Build # 9410 - Still Failing!

Posted by Mark Miller <ma...@gmail.com>.
Fail is:

/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8

- Mark

http://about.me/markrmiller

On Feb 18, 2014, at 12:00 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9410/
> Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
> 
> All tests passed
> 
> Build Log:
> [...truncated 28008 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:64: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8
> 
> Total time: 58 minutes 12 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60-ea-b04) - Build # 9410 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9410/
Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 28008 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:64: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8

Total time: 58 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/ibm-j9-jdk7) - Build # 9409 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9409/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

4 tests failed.
REGRESSION:  org.apache.solr.handler.component.DistributedQueryElevationComponentTest.testDistribSearch

Error Message:
.responseHeader.params.rows!=version (unordered or missing)

Stack Trace:
junit.framework.AssertionFailedError: .responseHeader.params.rows!=version (unordered or missing)
	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:47AB96ABC008AA4D]:0)
	at junit.framework.Assert.fail(Assert.java:50)
	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.DistributedQueryElevationComponentTest.doTest(DistributedQueryElevationComponentTest.java:81)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.handler.component.TermVectorComponentDistributedTest.testDistribSearch

Error Message:
.termVectors.0.test_basictv!=test_offtv (unordered or missing)

Stack Trace:
junit.framework.AssertionFailedError: .termVectors.0.test_basictv!=test_offtv (unordered or missing)
	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:47AB96ABC008AA4D]:0)
	at junit.framework.Assert.fail(Assert.java:50)
	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.TermVectorComponentDistributedTest.doTest(TermVectorComponentDistributedTest.java:137)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:A7D8C2535F86C087]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded(SignatureUpdateProcessorFactoryTest.java:222)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection

Error Message:
expected:<1> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<3>
	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:9D7DE37BB410690D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)




Build Log:
[...truncated 10647 lines...]
   [junit4] Suite: org.apache.solr.handler.component.TermVectorComponentDistributedTest
   [junit4]   2> 1228882 T3720 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1228883 T3720 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1228885 T3720 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1228887 T3720 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1228888 T3720 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49891
   [junit4]   2> 1228889 T3720 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1228889 T3720 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1228890 T3720 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 1228890 T3720 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 1228899 T3720 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 1228932 T3720 oasc.CoreContainer.<init> New CoreContainer -1567636093
   [junit4]   2> 1228933 T3720 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 1228934 T3720 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1228935 T3720 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1228935 T3720 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1228936 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1228937 T3720 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1228937 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1228938 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1228938 T3720 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1228939 T3720 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1228942 T3720 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1228942 T3720 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1228943 T3720 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1228946 T3730 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 1228947 T3730 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 1228947 T3730 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 1228947 T3730 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 1228984 T3730 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1229034 T3730 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1229035 T3730 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1229041 T3730 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1229376 T3730 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1229377 T3730 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1229378 T3730 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1229382 T3730 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1229385 T3730 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1229390 T3730 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1229393 T3730 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1229396 T3730 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1229397 T3730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1229397 T3730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1229398 T3730 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1229398 T3730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1229398 T3730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1229399 T3730 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1229399 T3730 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/
   [junit4]   2> 1229400 T3730 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48da44ba
   [junit4]   2> 1229400 T3730 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data
   [junit4]   2> 1229401 T3730 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index/
   [junit4]   2> 1229401 T3730 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1229401 T3730 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index
   [junit4]   2> 1229402 T3730 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=75.07421875, floorSegmentMB=1.3525390625, forceMergeDeletesPctAllowed=20.919694207225035, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1229403 T3730 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21ab0515 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47fa2a18)),segFN=segments_1,generation=1}
   [junit4]   2> 1229403 T3730 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1229403 T3730 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 1229405 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1229406 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1229406 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 1229406 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 1229407 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1229407 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1229407 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 1229408 T3730 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 1229408 T3730 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 1229408 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 1229409 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1229409 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1229410 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1229410 T3730 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1229410 T3730 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1229410 T3730 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1229411 T3730 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1229411 T3730 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1229412 T3730 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 1229412 T3730 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 1229413 T3730 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1229413 T3730 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1229413 T3730 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1229414 T3730 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1229414 T3730 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1229415 T3730 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1229415 T3730 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229415 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229416 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229416 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229417 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229417 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229417 T3730 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229418 T3730 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229418 T3730 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 1229419 T3730 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 1229420 T3730 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1229421 T3730 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 1229423 T3730 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1229424 T3730 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1229424 T3730 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6150124125500037]
   [junit4]   2> 1229425 T3730 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21ab0515 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47fa2a18)),segFN=segments_1,generation=1}
   [junit4]   2> 1229425 T3730 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1229426 T3730 oass.SolrIndexSearcher.<init> Opening Searcher@6ee1c17[collection1] main
   [junit4]   2> 1229426 T3730 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 1229479 T3730 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1229496 T3731 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ee1c17[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1229498 T3730 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1229499 T3720 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1229500 T3720 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1229501 T3720 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1229503 T3720 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40877
   [junit4]   2> 1229503 T3720 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1229504 T3720 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1229505 T3720 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 1229505 T3720 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 1229513 T3720 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 1229547 T3720 oasc.CoreContainer.<init> New CoreContainer -796612920
   [junit4]   2> 1229547 T3720 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 1229549 T3720 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1229549 T3720 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1229550 T3720 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1229550 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1229551 T3720 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1229551 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1229552 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1229553 T3720 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1229553 T3720 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1229556 T3720 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1229557 T3720 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1229557 T3720 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1229561 T3742 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 1229561 T3742 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 1229562 T3742 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 1229562 T3742 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 1229598 T3742 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1229649 T3742 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1229650 T3742 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1229656 T3742 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1230002 T3742 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1230002 T3742 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1230003 T3742 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1230008 T3742 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1230010 T3742 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1230015 T3742 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1230019 T3742 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1230023 T3742 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1230023 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1230024 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1230024 T3742 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1230024 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1230025 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1230025 T3742 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1230025 T3742 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/
   [junit4]   2> 1230026 T3742 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48da44ba
   [junit4]   2> 1230027 T3742 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data
   [junit4]   2> 1230027 T3742 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index/
   [junit4]   2> 1230027 T3742 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1230028 T3742 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index
   [junit4]   2> 1230028 T3742 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=75.07421875, floorSegmentMB=1.3525390625, forceMergeDeletesPctAllowed=20.919694207225035, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1230029 T3742 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1ac2fb29 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15c91418)),segFN=segments_1,generation=1}
   [junit4]   2> 1230029 T3742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1230030 T3742 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 1230032 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1230032 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1230033 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 1230033 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 1230033 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1230034 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1230034 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 1230034 T3742 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 1230035 T3742 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 1230035 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 1230035 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1230036 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1230036 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1230036 T3742 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1230037 T3742 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1230037 T3742 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1230038 T3742 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1230038 T3742 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1230039 T3742 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 1230039 T3742 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 1230040 T3742 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1230040 T3742 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1230041 T3742 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1230041 T3742 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1230041 T3742 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1230042 T3742 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1230042 T3742 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230043 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230043 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230044 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230044 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230045 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230045 T3742 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230046 T3742 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230046 T3742 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 1230047 T3742 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 1230048 T3742 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1230049 T3742 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 1230051 T3742 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1230052 T3742 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1230053 T3742 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6150124125500037]
   [junit4]   2> 1230053 T3742 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1ac2fb29 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15c91418)),segFN=segments_1,generation=1}
   [junit4]   2> 1230053 T3742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1230054 T3742 oass.SolrIndexSearcher.<init> Opening Searcher@91d000da[collection1] main
   [junit4]   2> 1230054 T3742 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 1230091 T3742 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1230152 T3743 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@91d000da[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1230156 T3742 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1230157 T3720 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1230157 T3720 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C2383 name=collection1 org.apache.solr.core.SolrCore@42a2a7d
   [junit4]   2> 1230163 T3727 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[0 (1460388493974831104)]} 0 4
   [junit4]   2> ASYNC  NEW_CORE C2384 name=collection1 org.apache.solr.core.SolrCore@fa0180bd
   [junit4]   2> 1230167 T3740 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[0 (1460388493982171136)]} 0 1
   [junit4]   2> 1230171 T3728 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1 (1460388493986365440)]} 0 1
   [junit4]   2> 1230175 T3741 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1 (1460388493990559744)]} 0 1
   [junit4]   2> 1230177 T3729 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1460388493993705472)]} 0 0
   [junit4]   2> 1230180 T3738 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[2 (1460388493996851200)]} 0 0
   [junit4]   2> 1230183 T3726 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1460388493998948352)]} 0 1
   [junit4]   2> 1230186 T3739 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[3 (1460388494002094080)]} 0 1
   [junit4]   2> 1230189 T3727 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1460388494005239808)]} 0 1
   [junit4]   2> 1230191 T3740 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1460388494008385536)]} 0 0
   [junit4]   2> 1230194 T3728 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[5 (1460388494011531264)]} 0 0
   [junit4]   2> 1230197 T3741 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1460388494014676992)]} 0 0
   [junit4]   2> 1230200 T3729 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[6 (1460388494017822720)]} 0 0
   [junit4]   2> 1230203 T3738 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1460388494019919872)]} 0 1
   [junit4]   2> 1230206 T3726 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7 (1460388494023065600)]} 0 1
   [junit4]   2> 1230210 T3739 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7 (1460388494028308480)]} 0 0
   [junit4]   2> 1230214 T3727 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[8 (1460388494032502784)]} 0 0
   [junit4]   2> 1230217 T3740 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1460388494034599936)]} 0 1
   [junit4]   2> 1230219 T3728 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1460388494037745664)]} 0 0
   [junit4]   2> 1230222 T3741 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1460388494039842816)]} 0 1
   [junit4]   2> 1230224 T3729 C2383 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1230228 T3729 C2383 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21ab0515 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47fa2a18)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21ab0515 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47fa2a18)),segFN=segments_2,generation=2}
   [junit4]   2> 1230229 T3729 C2383 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 1230233 T3729 C2383 oass.SolrIndexSearcher.<init> Opening Searcher@b39fe3d5[collection1] main
   [junit4]   2> 1230234 T3729 C2383 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1230236 T3731 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b39fe3d5[collection1] main{StandardDirectoryReader(segments_2:3:nrt _0(4.8):c10)}
   [junit4]   2> 1230236 T3729 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={commit=true&softCommit=false&version=2&wt=javabin&waitSearcher=true} {commit=} 0 12
   [junit4]   2> 1230238 T3738 C2384 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1230242 T3738 C2384 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1ac2fb29 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15c91418)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1ac2fb29 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15c91418)),segFN=segments_2,generation=2}
   [junit4]   2> 1230243 T3738 C2384 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 1230245 T3738 C2384 oass.SolrIndexSearcher.<init> Opening Searcher@24fe01a2[collection1] main
   [junit4]   2> 1230246 T3738 C2384 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1230247 T3743 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24fe01a2[collection1] main{StandardDirectoryReader(segments_2:3:nrt _0(4.8):c10)}
   [junit4]   2> 1230247 T3738 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&waitSearcher=true&commit=true&softCommit=false} {commit=} 0 9
   [junit4]   2> 1230256 T3726 C2383 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=id:0&shards.qt=tvrh&tv.tf=true&tv=true&qt=tvrh&sort=id+desc&version=2&distrib=false&wt=javabin} hits=1 status=0 QTime=6 
   [junit4]   2> 1230260 T3740 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&sort=id+desc&wt=javabin&rows=10&NOW=1392734998778&q=id:0&fl=id&tv=true&qt=tvrh&tv.tf=true&fsv=true&isShard=true&distrib=false&start=0&shard.url=http://127.0.0.1:40877/|[ff01::114]:33332/&shards.qt=tvrh} hits=1 status=0 QTime=0 
   [junit4]   2> 1230263 T3741 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&isShard=true&qt=tvrh&tv=true&tv.tf=true&wt=javabin&distrib=false&NOW=1392734998778&ids=0&shards.qt=tvrh&q=id:0&shard.url=http://127.0.0.1:40877/|[ff01::114]:33332/} status=0 QTime=0 
   [junit4]   2> 1230264 T3739 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shards=http://127.0.0.1:40877/|[ff01::114]:33332/&q=id:0&shards.qt=tvrh&tv=true&qt=tvrh&tv.tf=true&version=2&sort=id+desc&wt=javabin} hits=1 status=0 QTime=6 
   [junit4]   2> 1230267 T3727 C2383 oasc.SolrCore.execute [collection1] webapp= path=/select params={distrib=false&tv.fl=test_basictv,test_offtv&wt=javabin&tv=true&qt=tvrh&tv.tf=true&version=2&sort=id+asc&q=id:0&shards.qt=tvrh&fl=*,score} hits=1 status=0 QTime=1 
   [junit4]   2> 1230271 T3740 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={wt=javabin&version=2&sort=id+asc&NOW=1392734998789&q=id:0&fl=id,score&rows=10&distrib=false&start=0&tv.fl=test_basictv,test_offtv&tv=true&qt=tvrh&tv.tf=true&fsv=true&isShard=true&shard.url=http://127.0.0.1:40877/&shards.qt=tvrh} hits=1 status=0 QTime=0 
   [junit4]   2> 1230274 T3741 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&wt=javabin&q=id:0&NOW=1392734998789&fl=*,score&isShard=true&tv=true&qt=tvrh&tv.tf=true&tv.fl=test_basictv,test_offtv&distrib=false&shards.qt=tvrh&ids=0&shard.url=http://127.0.0.1:40877/} status=0 QTime=0 
   [junit4]   2> 1230275 T3738 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shards=http://127.0.0.1:40877/&q=id:0&shards.qt=tvrh&fl=*,score&tv.fl=test_basictv,test_offtv&wt=javabin&version=2&sort=id+asc&tv=true&qt=tvrh&tv.tf=true} hits=1 status=0 QTime=6 
   [junit4]   2> 1230278 T3728 C2383 oasc.SolrCore.execute [collection1] webapp= path=/select params={wt=javabin&tv.fl=test_basictv&tv.fl=test_offtv&distrib=false&sort=id+asc&version=2&tv.tf=true&tv=true&qt=tvrh&fl=*,score&q=id:0&shards.qt=tvrh} hits=1 status=0 QTime=1 
   [junit4]   2> 1230284 T3740 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shard.url=http://127.0.0.1:40877/|[ff01::114]:33332/|[ff01::213]:33332/&shards.qt=tvrh&start=0&distrib=false&tv.fl=test_basictv&tv.fl=test_offtv&tv.tf=true&fsv=true&tv=true&qt=tvrh&isShard=true&fl=id,score&NOW=1392734998800&q=id:0&rows=10&wt=javabin&sort=id+asc&version=2} hits=1 status=0 QTime=0 
   [junit4]   2> 1230287 T3741 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shards.qt=tvrh&ids=0&shard.url=http://127.0.0.1:40877/|[ff01::114]:33332/|[ff01::213]:33332/&tv.fl=test_basictv&tv.fl=test_offtv&distrib=false&isShard=true&tv.tf=true&tv=true&qt=tvrh&fl=*,score&q=id:0&NOW=1392734998800&wt=javabin&version=2} status=0 QTime=1 
   [junit4]   2> 1230288 T3739 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shards=http://127.0.0.1:40877/|[ff01::114]:33332/|[ff01::213]:33332/&q=id:0&shards.qt=tvrh&fl=*,score&tv.fl=test_basictv&tv.fl=test_offtv&wt=javabin&version=2&sort=id+asc&tv=true&qt=tvrh&tv.tf=true} hits=1 status=0 QTime=8 
   [junit4]   2> 1230289 T3720 oas.BaseDistributedSearchTestCase.compareSolrResponses ERROR Mismatched responses:
   [junit4]   2> 	{responseHeader={status=0,QTime=8},response={numFound=1,start=0,maxScore=2.609438,docs=[SolrDocument{id=0, range_facet_si=0, range_facet_l=[0], range_facet_sl=[0], test_posofftv=[This is a title and another title], test_basictv=[This is a title and another title], test_notv=[This is a title and another title], test_postv=[This is a title and another title], test_offtv=[This is a title and another title], _version_=1460388493982171136, intDefault=42, timestamp=Tue Feb 18 05:49:58 AKST 2014, multiDefault=[muLti-Default], score=2.609438}]},termVectors={uniqueKeyFieldName=id,0={uniqueKey=0,test_basictv={anoth={tf=1},titl={tf=2}},test_offtv={anoth={tf=1},titl={tf=2}}}}}
   [junit4]   2> 	{responseHeader={status=0,QTime=1},response={numFound=1,start=0,maxScore=2.609438,docs=[SolrDocument{id=0, range_facet_si=0, range_facet_l=[0], range_facet_sl=[0], test_posofftv=[This is a title and another title], test_basictv=[This is a title and another title], test_notv=[This is a title and another title], test_postv=[This is a title and another title], test_offtv=[This is a title and another title], _version_=1460388493974831104, intDefault=42, timestamp=Tue Feb 18 05:49:58 AKST 2014, multiDefault=[muLti-Default], score=2.609438}]},termVectors={uniqueKeyFieldName=id,0={uniqueKey=0,test_offtv={anoth={tf=1},titl={tf=2}},test_basictv={anoth={tf=1},titl={tf=2}}}}}
   [junit4]   2> 1230310 T3720 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1567636093
   [junit4]   2> 1230311 T3720 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@42a2a7d
   [junit4]   2> 1230321 T3720 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=1285,transaction_logs_total_number=1}
   [junit4]   2> 1230322 T3720 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1230323 T3720 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1230324 T3720 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1230325 T3720 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1230326 T3720 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1230327 T3720 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index;done=false>>]
   [junit4]   2> 1230328 T3720 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index
   [junit4]   2> 1230328 T3720 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data;done=false>>]
   [junit4]   2> 1230329 T3720 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data
   [junit4]   2> 1230329 T3720 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1230402 T3720 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-796612920
   [junit4]   2> 1230403 T3720 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@fa0180bd
   [junit4]   2> 1230417 T3720 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=1285,transaction_logs_total_number=1}
   [junit4]   2> 1230418 T3720 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1230419 T3720 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1230424 T3720 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1230425 T3720 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1230426 T3720 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1230427 T3720 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data;done=false>>]
   [junit4]   2> 1230428 T3720 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data
   [junit4]   2> 1230429 T3720 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index;done=false>>]
   [junit4]   2> 1230430 T3720 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index
   [junit4]   2> 1230431 T3720 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1230489 T3720 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TermVectorComponentDistributedTest -Dtests.method=testDistribSearch -Dtests.seed=C64D18B3B757CA71 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_RS_#Latn -Dtests.timezone=America/Yakutat -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.61s J1 | TermVectorComponentDistributedTest.testDistribSearch <<<
   [junit4]    > Throwable #1: junit.framework.AssertionFailedError: .termVectors.0.test_basictv!=test_offtv (unordered or missing)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:47AB96ABC008AA4D]:0)
   [junit4]    > 	at junit.framework.Assert.fail(Assert.java:50)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
   [junit4]    > 	at org.apache.solr.handler.component.TermVectorComponentDistributedTest.doTest(TermVectorComponentDistributedTest.java:137)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 1230498 T3720 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene46: {test_posofftv=PostingsFormat(name=FSTPulsing41), timestamp=PostingsFormat(name=MockSep), _version_=PostingsFormat(name=FSTPulsing41), range_facet_l=PostingsFormat(name=MockSep), intDefault=PostingsFormat(name=FSTPulsing41), id=PostingsFormat(name=SimpleText), multiDefault=PostingsFormat(name=MockSep), test_offtv=PostingsFormat(name=FSTPulsing41), range_facet_si=PostingsFormat(name=MockSep), test_postv=Lucene41(blocksize=128), range_facet_sl=PostingsFormat(name=SimpleText), test_basictv=PostingsFormat(name=SimpleText), test_notv=PostingsFormat(name=MockSep)}, docValues:{}, sim=DefaultSimilarity, locale=sr_RS_#Latn, timezone=America/Yakutat
   [junit4]   2> NOTE: Linux 3.8.0-35-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=166791848,total=295305216
   [junit4]   2> NOTE: All tests run in this JVM: [RemoteQueryErrorTest, TestRTGBase, ParsingFieldUpdateProcessorsTest, IndexSchemaRuntimeFieldTest, TestNRTOpen, OverseerRolesTest, DocumentAnalysisRequestHandlerTest, TestDynamicFieldResource, TestFieldCollectionResource, TimeZoneUtilsTest, TestStressRecovery, TestCopyFieldCollectionResource, TestStressReorder, TestDistributedSearch, SpellingQueryConverterTest, TestAtomicUpdateErrorCases, HdfsCollectionsAPIDistributedZkTest, NumericFieldsTest, TestJmxIntegration, TestFieldTypeResource, DebugComponentTest, TestFiltering, TestSolrCoreProperties, AddBlockUpdateTest, TestSchemaVersionResource, TestQuerySenderNoQuery, TestCodecSupport, DefaultValueUpdateProcessorTest, CurrencyFieldXmlFileTest, TestReload, HighlighterTest, SortByFunctionTest, MigrateRouteKeyTest, TestFreeTextSuggestions, TestNumberUtils, BasicDistributedZk2Test, CurrencyFieldOpenExchangeTest, TestMultiCoreConfBootstrap, TestCoreDiscovery, TestCollationFieldDocValues, TestNonDefinedSimilarityFactory, TestSolrXmlPersistor, StatsComponentTest, SolrPluginUtilsTest, DistanceFunctionTest, FieldAnalysisRequestHandlerTest, SuggesterWFSTTest, BadIndexSchemaTest, CursorPagingTest, SpellCheckComponentTest, TestHashPartitioner, OpenExchangeRatesOrgProviderTest, TestArbitraryIndexDir, TestSchemaSimilarityResource, BJQParserTest, TestSimpleQParserPlugin, TestCollationField, ZkNodePropsTest, XsltUpdateRequestHandlerTest, RequiredFieldsTest, DocValuesMultiTest, TestSweetSpotSimilarityFactory, TestManagedSchemaFieldResource, PreAnalyzedFieldTest, HardAutoCommitTest, UniqFieldsUpdateProcessorFactoryTest, TestUpdate, TestDefaultSearchFieldResource, TestIndexSearcher, DirectUpdateHandlerTest, TestFuzzyAnalyzedSuggestions, SolrRequestParserTest, TestHighlightDedupGrouping, TestIndexingPerformance, SuggestComponentTest, TestStressUserVersions, FileUtilsTest, TestFoldingMultitermQuery, TestFileDictionaryLookup, ConvertedLegacyTest, TestHighFrequencyDictionaryFactory, TestOmitPositions, TestZkChroot, PathHierarchyTokenizerFactoryTest, CursorMarkTest, TestLazyCores, PingRequestHandlerTest, TestClassNameShortening, NotRequiredUniqueKeyTest, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, BasicZkTest, ClusterStateUpdateTest, TestRandomFaceting, TestFaceting, TestRecovery, TermVectorComponentDistributedTest]
   [junit4] Completed on J1 in 1.64s, 1 test, 1 failure <<< FAILURES!

[...truncated 43 lines...]
   [junit4] Suite: org.apache.solr.handler.component.DistributedQueryElevationComponentTest
   [junit4]   2> 1326592 T4701 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1326592 T4701 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nvic/s
   [junit4]   2> 1326596 T4701 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1326597 T4701 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1326600 T4701 oejus.SslContextFactory.doStart Enabled Protocols [TLSv1] of [SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1326604 T4701 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60904
   [junit4]   2> 1326606 T4701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1326607 T4701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1326607 T4701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 1326608 T4701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 1326616 T4701 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 1326650 T4701 oasc.CoreContainer.<init> New CoreContainer 1078884461
   [junit4]   2> 1326651 T4701 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 1326652 T4701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1326653 T4701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 1326653 T4701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1326654 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1326654 T4701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1326655 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1326655 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1326656 T4701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1326657 T4701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1326661 T4701 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1326662 T4701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1326662 T4701 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1326666 T4711 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 1326666 T4711 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 1326667 T4711 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 1326667 T4711 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 1326698 T4711 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1326751 T4711 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
   [junit4]   2> 1326752 T4711 oass.IndexSchema.readSchema Reading Solr Schema from schema11.xml
   [junit4]   2> 1326756 T4711 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 1326891 T4711 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326895 T4711 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326899 T4711 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326900 T4711 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data/
   [junit4]   2> 1326900 T4711 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad68826
   [junit4]   2> 1326901 T4711 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data
   [junit4]   2> 1326902 T4711 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data/index/
   [junit4]   2> 1326902 T4711 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326903 T4711 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data/index
   [junit4]   2> 1326903 T4711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=13, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1326904 T4711 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2fc35f47 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@dc98fc9e),segFN=segments_1,generation=1}
   [junit4]   2> 1326904 T4711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326906 T4711 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1326907 T4711 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1326907 T4711 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1326908 T4711 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1326909 T4711 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1326910 T4711 oasc.RequestHandlers.initHandlersFromConfig WARN no default request handler is registered (either '/select' or 'standard')
   [junit4]   2> 1326911 T4711 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 1326912 T4711 oasc.SolrCore.initDeprecatedSupport WARN adding ShowFileRequestHandler with hidden files: [SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-TRANSFORMERS.XML, SCHEMA-REQUIRED-FIELDS.XML, DA_UTF8.XML, HUNSPELL-TEST.AFF, CURRENCY.XML, STOPWITHBOM.TXT, SCHEMA-DOCVALUES.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-POSTINGSHIGHLIGHT.XML, BAD-SCHEMA-MISPLACED-ASTERISK-COPYFIELD-DEST-SHOULD-FAIL-TEST.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, SCHEMA-BM25.XML, BAD-SCHEMA-CURRENCY-FT-BOGUS-CODE-IN-XML.XML, SCHEMA-LMJELINEKMERCER.XML, FREETEXTSUGGEST.TXT, SOLRCONFIG_CODEC.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, KEEP-1.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, BAD-MP-SOLRCONFIG.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-FUNCTIONQUERY.XML, BAD-SOLRCONFIG-MULTIPLE-CFS.XML, SOLRCONFIG-HIGHLIGHT.XML, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, ENUMSCONFIG.XML, SOLRCONFIG-ELEVATE.XML, SCHEMA-CUSTOM-FIELD.XML, SCHEMA-MINIMAL.XML, SOLRCONFIG-MERGEPOLICY-DEFAULTS.XML, XSLT, SCHEMA-ADD-SCHEMA-FIELDS-UPDATE-PROCESSOR.XML, SYNONYMS.TXT, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-MASTER2.XML, REGEX-BOOST-PROCESSOR-TEST.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, BAD-SCHEMA-SWEETSPOT-PARTIAL-HYPERBOLIC.XML, STOP-2.TXT, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SOLRCONFIG-MERGEPOLICY-LEGACY.XML, BAD-SOLRCONFIG-MULTIPLE-INDEXCONFIGS.XML, MAPPING-ISOLATIN1ACCENT.TXT, SCHEMA-TFIDF.XML, STOPTYPES-1.TXT, SCHEMA-SIMPLEQPPLUGIN.XML, BAD-SCHEMA-DYNAMICFIELD-DEFAULT-VAL.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SOLRCONFIG-MASTER.XML, SCHEMA-DISTRIBUTED-MISSING-SORT.XML, BAD-CURRENCY.XML, SOLRCONFIG-IMPLICITPROPERTIES.XML, SCHEMA11.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REST.XML, SCHEMA-COLLATE.XML, SCHEMA-SIM.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML, SOLRCONFIG-REPEATER.XML, JASUGGEST.TXT, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-TEST-MISC.XML, SCHEMA-SWEETSPOT.XML, SOLRCONFIG-MANAGED-SCHEMA.XML, SOLRCONFIG-DEEPPAGING.XML, SOLRCONFIG-REQHANDLER.INCL, BAD-SCHEMA-MULTIPLE-ASTERISK-COPYFIELD-DEST-SHOULD-FAIL-TEST.XML, STOPTYPES-2.TXT, SOLRCONFIG-SOLR-749.XML, BAD-SCHEMA-MISPLACED-ASTERISK-COPYFIELD-SOURCE-SHOULD-FAIL-TEST.XML, SOLRCONFIG-QUERY-PARSER-INIT.XML, SCHEMA-DOCVALUESMULTI.XML, SOLRCONFIG-CACHING.XML, BAD-SOLRCONFIG-MANAGED-SCHEMA-NAMED-SCHEMA.XML.XML, SOLRCONFIG-DEFAULTS.XML, BAD-SCHEMA-DYNAMICFIELD-REQUIRED.XML, HUNSPELL-TEST.DIC, SCHEMA-SORTS.XML, SOLRCONFIG.SNIPPET.RANDOMINDEXCONFIG.XML, BAD-SOLRCONFIG-SCHEMA-MUTABLE-BUT-NOT-MANAGED.XML, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, STOP-1.TXT, SCHEMA-DOCVALUESMISSING.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-WARMER.XML, ADDFIELDS.UPDATEPROCESSOR.JS, TRIVIAL.UPDATEPROCESSOR0.JS, SCHEMA-SPELLCHECKER.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMASURROUND.XML, SOLRCONFIG-MASTER1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, SCHEMA-REPLICATION2.XML, BAD-SCHEMA-BOGUS-FIELD-PARAMETERS.XML, BAD-SCHEMA-SWEETSPOT-PARTIAL-BASELINE.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-CURRENCY-FT-OER-NORATES.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-MASTER3.XML, SOLRCONFIG-NOOPREGEN.XML, WDFTYPES.TXT, STOPWORDS.TXT, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-DFR.XML, OPEN-EXCHANGE-RATES.JSON, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SOLRCONFIG-TIEREDMERGEPOLICY.XML, SCHEMA-CUSTOMFIELD.XML, SCHEMA_CODEC.XML, SCHEMA-XINCLUDE.XML, SCHEMA-DOCVALUESFACETING.XML, SCHEMA-SYNONYM-TOKENIZER.XML, SCHEMA-POSTINGSHIGHLIGHT.XML, SCHEMA-SNIPPET-TYPE.XML, SCHEMA-SPATIAL.XML, ANALYZINGINFIXSUGGEST.TXT, SCHEMA-ID-AND-VERSION-FIELDS-ONLY.XML, OLD_SYNONYMS.TXT, SCHEMA-COLLATE-DV.XML, SCHEMA-NUMERIC.XML, SOLRCONFIG-EXTERNALVERSIONCONSTRAINT.XML, SOLRCONFIG-PARSING-UPDATE-PROCESSOR-CHAINS.XML, SOLRCONFIG-COMPONENTS-NAME.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-LOGMERGEPOLICY.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-TINY.XML, PROTWORDS.TXT, SOLRCONFIG-TLOG.XML, SCHEMA-BEHAVIOR.XML, SCHEMA-FOLDING.XML, SOLRCONFIG-SPELLCHECKER.XML, COMPOUNDDICTIONARY.TXT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-BASIC.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SOLRCONFIG_PERF.XML, STOP-SNOWBALL.TXT, SOLRCONFIG-SUGGESTERCOMPONENT.XML, BAD-SCHEMA-BOGUS-ANALYSIS-PARAMETERS.XML, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, BAD-SOLRCONFIG-MULTIPLE-DIRFACTORY.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, BAD-SCHEMA-MULTIPLE-ASTERISK-COPYFIELD-SOURCE-SHOULD-FAIL-TEST.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, BAD-SOLRCONFIG-WARMER-NO-REOPEN.XML, SCHEMA-CLASS-NAME-SHORTENING-ON-SERIALIZATION.XML, SOLRCONFIG-DELPOLICY2.XML, DA_COMPOUNDDICTIONARY.TXT, BAD-SCHEMA-SWEETSPOT-PARTIAL-NORMS.XML, SOLRCONFIG-MINIMAL.XML, BAD_SOLRCONFIG.XML, FUZZYSUGGEST.TXT, SCHEMA-LUCENEMATCHVERSION.XML, SOLRCONFIG-SLAVE.XML, SCHEMA-IB.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-COLLATEFILTER.XML, SCHEMA15.XML, SCHEMA-ONE-FIELD-NO-DYNAMIC-FIELD-UNIQUE-KEY.XML, SOLRCONFIG-INFOSTREAM-LOGGING.XML, SOLRCONFIG-COLLAPSEQPARSER.XML, SCHEMA-SNIPPET-TYPES.INCL, STOPWORDSWRONGENCODING.TXT, BAD-SCHEMA-NON-GLOB-COPYFIELD-SOURCE-MATCHING-NOTHING-SHOULD-FAIL-TEST.XML, BAD-SCHEMA-CURRENCY-FT-BOGUS-DEFAULT-CODE.XML, SCHEMA-PHRASESUGGEST.XML, SOLRCONFIG-PHRASESUGGEST.XML, SCHEMA-REVERSED.XML, PHRASESUGGEST.TXT, SCHEMA-EFF.XML, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SCHEMA-SNIPPET-FIELD.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-XINCLUDE.XML, BAD-SCHEMA-SWEETSPOT-BOTH-TF.XML, FRENCHARTICLES.TXT, BLENDEDINFIXSUGGEST.TXT, KEEP-2.TXT, SCHEMA-ONE-FIELD-NO-DYNAMIC-FIELD.XML, SOLRCONFIG-QUERYSENDER.XML, SCHEMA-REST-LUCENE-MATCH-VERSION.XML, STEMDICT.TXT, SCHEMA-STOP-KEEP.XML, SOLRCONFIG-ADD-SCHEMA-FIELDS-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA-ENUMS.XML, BAD-SOLRCONFIG-UNEXPECTED-SCHEMA-ATTRIBUTE.XML, ELEVATE.XML, BAD-SCHEMA-UNSUPPORTED-DOCVALUES.XML, SCHEMA-TRIE.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-ALTDIRECTORY.XML, SCHEMA12.XML, HYPHENATION.DTD]
   [junit4]   2> 1326914 T4711 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1326914 T4711 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1326915 T4711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=97.2216796875, floorSegmentMB=1.3271484375, forceMergeDeletesPctAllowed=7.240384447564098, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1326916 T4711 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2fc35f47 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@dc98fc9e),segFN=segments_1,generation=1}
   [junit4]   2> 1326916 T4711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326917 T4711 oass.SolrIndexSearcher.<init> Opening Searcher@19aee66a[collection1] main
   [junit4]   2> 1326917 T4711 oashc.QueryElevationComponent.inform Loading QueryElevation from: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
   [junit4]   2> 1326927 T4711 oashc.QueryElevationComponent.inform Loading QueryElevation from: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
   [junit4]   2> 1326937 T4712 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19aee66a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1326938 T4711 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1326939 T4701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1326939 T4701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1326941 T4701 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1326943 T4701 oejus.SslContextFactory.doStart Enabled Protocols [TLSv1] of [SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1326944 T4701 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:43841
   [junit4]   2> 1326946 T4701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1326946 T4701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1326947 T4701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 1326948 T4701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 1326955 T4701 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 1326990 T4701 oasc.CoreContainer.<init> New CoreContainer 1655269720
   [junit4]   2> 1326990 T4701 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 1326992 T4701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1326992 T4701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 1326993 T4701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1326994 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1326994 T4701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1326995 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1326995 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1326996 T4701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1326996 T4701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1327001 T4701 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1327002 T4701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1327003 T4701 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1327006 T4723 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 1327006 T4723 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 1327007 T4723 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.

[...truncated too long message...]

ache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 1331100 T4757 oas.SolrTestCaseJ4.setUp ###Starting testNonStringFieldsValues
   [junit4]   2> 1331100 T4757 C2079 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1331101 T4757 C2079 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 1331103 T4757 C2079 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1331104 T4757 C2079 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@2fa85e79 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@963e4408)),segFN=segments_12,generation=38}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@2fa85e79 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@963e4408)),segFN=segments_13,generation=39}
   [junit4]   2> 1331105 T4757 C2079 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 39
   [junit4]   2> 1331106 T4757 C2079 oass.SolrIndexSearcher.<init> Opening Searcher@d08754da[collection1] main
   [junit4]   2> 1331107 T4757 C2079 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1331108 T4759 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d08754da[collection1] main{StandardDirectoryReader(segments_13:163:nrt)}
   [junit4]   2> 1331109 T4757 C2079 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 6
   [junit4]   2> 1331112 T4757 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1331114 T4757 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1331116 T4757 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@2fa85e79 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@963e4408)),segFN=segments_13,generation=39}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@2fa85e79 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@963e4408)),segFN=segments_14,generation=40}
   [junit4]   2> 1331116 T4757 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 40
   [junit4]   2> 1331118 T4757 oass.SolrIndexSearcher.<init> Opening Searcher@cbb6d722[collection1] main
   [junit4]   2> 1331118 T4757 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1331121 T4759 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@cbb6d722[collection1] main{StandardDirectoryReader(segments_14:165:nrt _1f(4.8):C5/1:delGen=1)}
   [junit4]   2> 1331121 T4757 oas.SolrTestCaseJ4.tearDown ###Ending testNonStringFieldsValues
   [junit4]   2> 1331123 T4757 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 1331124 T4757 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=928948566
   [junit4]   2> 1331124 T4757 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@ab3e2629
   [junit4]   2> 1331137 T4757 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=42,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=253,cumulative_deletesById=0,cumulative_deletesByQuery=6,cumulative_errors=0}
   [junit4]   2> 1331138 T4757 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1331138 T4757 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1331139 T4757 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1331139 T4757 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1331140 T4757 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1331141 T4757 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833/index;done=false>>]
   [junit4]   2> 1331141 T4757 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833/index
   [junit4]   2> 1331142 T4757 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833;done=false>>]
   [junit4]   2> 1331142 T4757 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=sr_ME, timezone=America/Goose_Bay
   [junit4]   2> NOTE: Linux 3.8.0-35-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=49574792,total=339607552
   [junit4]   2> NOTE: All tests run in this JVM: [OverseerCollectionProcessorTest, TestCloudManagedSchemaAddField, SoftAutoCommitTest, TestWriterPerf, SOLR749Test, LeaderElectionTest, TestMergePolicyConfig, TermVectorComponentTest, TermsComponentTest, TestSystemIdResolver, TestElisionMultitermQuery, ResponseLogComponentTest, TestExtendedDismaxParser, TestDocumentBuilder, ShowFileRequestHandlerTest, TestBM25SimilarityFactory, HdfsLockFactoryTest, TestDistribDocBasedVersion, TestFieldResource, QueryResultKeyTest, FastVectorHighlighterTest, JsonLoaderTest, ExternalFileFieldSortTest, TestRandomDVFaceting, SuggesterTest, TestBadConfig, HdfsUnloadDistributedZkTest, PeerSyncTest, SimpleFacetsTest, QueryParsingTest, MBeansHandlerTest, TestLFUCache, DeleteShardTest, DistributedQueryComponentCustomSortTest, SpellPossibilityIteratorTest, DateMathParserTest, TestCollationKeyRangeQueries, ShardSplitTest, CachingDirectoryFactoryTest, TestRecoveryHdfs, TestInfoStreamLogging, CoreAdminHandlerTest, UpdateRequestProcessorFactoryTest, TestLRUCache, TestNonNRTOpen, TestLMJelinekMercerSimilarityFactory, BasicFunctionalityTest, TestReplicationHandler, ShardRoutingTest, TestFieldTypeCollectionResource, HdfsSyncSliceTest, TestOverriddenPrefixQueryForCustomFieldType, ReturnFieldsTest, TestSolrQueryParserResource, SliceStateUpdateTest, TestSolrXml, TestSchemaResource, TestRemoteStreaming, TestSearchPerf, TestPostingsSolrHighlighter, TestSchemaNameResource, BlockCacheTest, RegexBoostProcessorTest, DirectUpdateHandlerOptimizeTest, SearchHandlerTest, PolyFieldTest, TestAnalyzeInfixSuggestions, TestSuggestSpellingConverter, TestStressVersions, TestGroupingSearch, AutoCommitTest, CoreMergeIndexesAdminHandlerTest, TestInitQParser, ScriptEngineTest, IndexSchemaTest, FileBasedSpellCheckerTest, AnalysisAfterCoreReloadTest, WordBreakSolrSpellCheckerTest, ChangedSchemaMergeTest, SliceStateTest, TestTrie, CopyFieldTest, ResourceLoaderTest, TestShardHandlerFactory, QueryElevationComponentTest, TestDynamicFieldCollectionResource, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, SyncSliceTest, LeaderElectionIntegrationTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ZkSolrClientTest, ZkCLITest, ShardRoutingCustomTest, TestDistributedGrouping, DistributedSpellCheckComponentTest, TestRealTimeGet, TestJoin, SolrCoreTest, SolrCmdDistributorTest, TestFunctionQuery, DistributedQueryElevationComponentTest, SimplePostToolTest, SignatureUpdateProcessorFactoryTest]
   [junit4] Completed on J0 in 1.66s, 6 tests, 2 failures <<< FAILURES!

[...truncated 451 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:452: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 362 suites, 1563 tests, 4 failures, 36 ignored (8 assumptions)

Total time: 55 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build # 9408 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9408/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseParallelGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([B275C9688947A547]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([B275C9688947A547]:0)




Build Log:
[...truncated 11210 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> 622015 T1619 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 622015 T1619 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 622019 T1619 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1392721576617
   [junit4]   2> 622019 T1619 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 622020 T1620 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 622120 T1619 oasc.ZkTestServer.run start zk server on port:52473
   [junit4]   2> 622121 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622220 T1626 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1076396 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622220 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622221 T1619 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 622228 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622230 T1628 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ce6123 name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622230 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622230 T1619 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 622236 T1619 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 622241 T1619 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 622244 T1619 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 622249 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 622250 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 622258 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 622258 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 622265 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 622266 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 622271 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 622271 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 622276 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 622277 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 622282 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 622283 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 622287 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 622288 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 622293 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 622294 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 622298 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 622299 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 622304 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 622305 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 622310 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 622310 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 622317 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622319 T1630 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ff2649 name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622319 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622407 T1619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 622410 T1619 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 622411 T1619 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60884
   [junit4]   2> 622413 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 622413 T1619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 622413 T1619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921
   [junit4]   2> 622413 T1619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/'
   [junit4]   2> 622423 T1619 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/solr.xml
   [junit4]   2> 622454 T1619 oasc.CoreContainer.<init> New CoreContainer 17460510
   [junit4]   2> 622454 T1619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/]
   [junit4]   2> 622455 T1619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 622456 T1619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 622456 T1619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 622456 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 622456 T1619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 622457 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 622457 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 622457 T1619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 622457 T1619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 622461 T1619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 622461 T1619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 622461 T1619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 622461 T1619 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52473/solr
   [junit4]   2> 622462 T1619 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 622463 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622464 T1641 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b0a870 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622464 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622466 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622467 T1643 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@534cae name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622467 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622468 T1619 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 622471 T1619 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 622475 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 622476 T1619 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60884_
   [junit4]   2> 622477 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60884_
   [junit4]   2> 622479 T1619 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 622481 T1619 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 622483 T1619 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60884_
   [junit4]   2> 622483 T1619 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 622485 T1619 oasc.Overseer.start Overseer (id=91273401245302788-127.0.0.1:60884_-n_0000000000) starting
   [junit4]   2> 622488 T1619 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 622492 T1645 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 622492 T1619 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 622493 T1645 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 622495 T1619 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 622497 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 622500 T1644 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 622502 T1646 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 622503 T1646 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 622504 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 622504 T1646 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 622505 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 622505 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60884",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60884_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 622506 T1644 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 622506 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 622507 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 622508 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 623504 T1646 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 623504 T1646 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1
   [junit4]   2> 623505 T1646 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 623505 T1646 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 623506 T1646 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 623507 T1646 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 623507 T1646 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1/'
   [junit4]   2> 623507 T1646 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1/lib/classes/' to classloader
   [junit4]   2> 623507 T1646 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1/lib/README' to classloader
   [junit4]   2> 623535 T1646 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 623565 T1646 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 623566 T1646 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 623572 T1646 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 623835 T1646 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 623837 T1646 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 623838 T1646 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 623842 T1646 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 623860 T1646 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 623866 T1646 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/
   [junit4]   2> 623866 T1646 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@121387e
   [junit4]   2> 623867 T1646 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data
   [junit4]   2> 623867 T1646 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index/
   [junit4]   2> 623867 T1646 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 623868 T1646 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index
   [junit4]   2> 623868 T1646 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=18.9658203125, floorSegmentMB=2.013671875, forceMergeDeletesPctAllowed=28.875788144137676, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3897341227826966
   [junit4]   2> 623872 T1646 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 623872 T1646 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 623874 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 623874 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 623874 T1646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 623875 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 623875 T1646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 623875 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 623875 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 623876 T1646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 623876 T1646 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 623876 T1646 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 623876 T1646 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 623877 T1646 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 623877 T1646 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 623877 T1646 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 623877 T1646 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 623878 T1646 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 623880 T1646 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 623882 T1646 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 623882 T1646 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 623883 T1646 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1000120485, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 623883 T1646 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 623883 T1646 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 623884 T1646 oass.SolrIndexSearcher.<init> Opening Searcher@a9fe08[collection1] main
   [junit4]   2> 623885 T1647 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a9fe08[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 623886 T1646 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 623887 T1619 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 623892 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 623891 T1650 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60884 collection:control_collection shard:shard1
   [junit4]   2> 623893 T1650 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 623894 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 623895 T1652 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69083c name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 623896 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 623897 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 623899 T1650 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 623899 T1619 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 623900 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 623901 T1650 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 623901 T1650 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C592 name=collection1 org.apache.solr.core.SolrCore@1a84cb9 url=https://127.0.0.1:60884/collection1 node=127.0.0.1:60884_ C592_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:60884, core=collection1, node_name=127.0.0.1:60884_}
   [junit4]   2> 623901 T1650 C592 P60884 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60884/collection1/
   [junit4]   2> 623902 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 623902 T1650 C592 P60884 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 623902 T1650 C592 P60884 oasc.SyncStrategy.syncToMe https://127.0.0.1:60884/collection1/ has no replicas
   [junit4]   2> 623902 T1650 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60884/collection1/ shard1
   [junit4]   2> 623903 T1650 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 623904 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 623907 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 623910 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 623971 T1619 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 623971 T1619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 623974 T1619 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 623975 T1619 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:55496
   [junit4]   2> 623976 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 623977 T1619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 623977 T1619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497
   [junit4]   2> 623977 T1619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/'
   [junit4]   2> 623987 T1619 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/solr.xml
   [junit4]   2> 624013 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 624014 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 624022 T1619 oasc.CoreContainer.<init> New CoreContainer 30476929
   [junit4]   2> 624022 T1619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/]
   [junit4]   2> 624023 T1619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 624023 T1619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 624024 T1619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 624024 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 624024 T1619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 624024 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 624025 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 624025 T1619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 624025 T1619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 624029 T1619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 624029 T1619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 624029 T1619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 624030 T1619 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52473/solr
   [junit4]   2> 624030 T1619 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 624031 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 624032 T1663 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13023e3 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 624032 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 624034 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 624035 T1665 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9c00ea name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 624036 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 624039 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 624058 T1650 oasc.ZkController.register We are https://127.0.0.1:60884/collection1/ and leader is https://127.0.0.1:60884/collection1/
   [junit4]   2> 624058 T1650 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60884
   [junit4]   2> 624059 T1650 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 624059 T1650 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 624059 T1650 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 624060 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 624060 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 624060 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 624060 T1650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 624062 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 624062 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60884",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60884_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 624064 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 624166 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 624166 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 624166 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 625042 T1619 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55496_
   [junit4]   2> 625044 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55496_
   [junit4]   2> 625048 T1665 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 625048 T1643 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 625048 T1652 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 625056 T1666 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 625056 T1666 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 625057 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 625057 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 625057 T1666 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 625058 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 625059 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 625059 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55496",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55496_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 625060 T1644 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 625060 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 625063 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 625165 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 625166 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 625166 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626058 T1666 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 626058 T1666 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1
   [junit4]   2> 626058 T1666 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 626059 T1666 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 626059 T1666 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 626060 T1666 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 626060 T1666 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1/'
   [junit4]   2> 626061 T1666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1/lib/classes/' to classloader
   [junit4]   2> 626061 T1666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1/lib/README' to classloader
   [junit4]   2> 626088 T1666 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 626119 T1666 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 626121 T1666 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 626126 T1666 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 626391 T1666 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 626393 T1666 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 626394 T1666 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 626397 T1666 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 626416 T1666 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 626416 T1666 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/
   [junit4]   2> 626416 T1666 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@121387e
   [junit4]   2> 626417 T1666 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1
   [junit4]   2> 626417 T1666 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index/
   [junit4]   2> 626417 T1666 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 626417 T1666 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index
   [junit4]   2> 626418 T1666 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=18.9658203125, floorSegmentMB=2.013671875, forceMergeDeletesPctAllowed=28.875788144137676, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3897341227826966
   [junit4]   2> 626420 T1666 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 626420 T1666 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 626422 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 626422 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 626423 T1666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 626423 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 626423 T1666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 626423 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 626424 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 626424 T1666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 626424 T1666 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 626424 T1666 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 626425 T1666 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 626425 T1666 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 626425 T1666 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 626426 T1666 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 626426 T1666 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 626426 T1666 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 626428 T1666 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 626430 T1666 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 626431 T1666 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 626431 T1666 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1000120485, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 626432 T1666 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 626432 T1666 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 626433 T1666 oass.SolrIndexSearcher.<init> Opening Searcher@ece25a[collection1] main
   [junit4]   2> 626435 T1667 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ece25a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 626437 T1666 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 626437 T1619 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 626438 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 626437 T1670 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:55496 collection:collection1 shard:shard3
   [junit4]   2> 626439 T1670 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 626444 T1670 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 626446 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626446 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626446 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626447 T1670 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 626447 T1670 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C593 name=collection1 org.apache.solr.core.SolrCore@c2a8ca url=https://127.0.0.1:55496/collection1 node=127.0.0.1:55496_ C593_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:55496, core=collection1, node_name=127.0.0.1:55496_}
   [junit4]   2> 626447 T1670 C593 P55496 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:55496/collection1/
   [junit4]   2> 626447 T1670 C593 P55496 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 626447 T1670 C593 P55496 oasc.SyncStrategy.syncToMe https://127.0.0.1:55496/collection1/ has no replicas
   [junit4]   2> 626447 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 626447 T1670 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:55496/collection1/ shard3
   [junit4]   2> 626448 T1670 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 626452 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626454 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626457 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626550 T1619 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 626551 T1619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 626554 T1619 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 626555 T1619 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36580
   [junit4]   2> 626557 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 626557 T1619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 626557 T1619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038
   [junit4]   2> 626557 T1619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/'
   [junit4]   2> 626559 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626559 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626559 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626569 T1619 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/solr.xml
   [junit4]   2> 626601 T1619 oasc.CoreContainer.<init> New CoreContainer 6419595
   [junit4]   2> 626601 T1619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/]
   [junit4]   2> 626602 T1619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 626602 T1619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 626603 T1619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 626603 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 626603 T1619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 626603 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 626604 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 626604 T1619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 626604 T1619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 626605 T1670 oasc.ZkController.register We are https://127.0.0.1:55496/collection1/ and leader is https://127.0.0.1:55496/collection1/
   [junit4]   2> 626605 T1670 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:55496
   [junit4]   2> 626605 T1670 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 626606 T1670 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 626606 T1670 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 626607 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626607 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626607 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626608 T1670 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 626608 T1619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 626609 T1619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 626609 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 626609 T1619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 626610 T1619 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52473/solr
   [junit4]   2> 626610 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55496",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55496_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 626610 T1619 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 626611 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 626612 T1681 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@906624 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 626612 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 626613 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626614 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 626615 T1683 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d631a name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 626615 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 626619 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 626716 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626716 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626716 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626716 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 627622 T1619 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36580_
   [junit4]   2> 627623 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36580_
   [junit4]   2> 627625 T1652 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 627625 T1643 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 627625 T1665 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 627625 T1683 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 627632 T1684 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 627633 T1684 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 627634 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 627634 T1684 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 627634 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 627634 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 627636 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 627636 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36580",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36580_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 627637 T1644 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 627637 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 627639 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 627741 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 627741 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 627741 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 627741 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 628634 T1684 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 628635 T1684 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1
   [junit4]   2> 628635 T1684 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 628635 T1684 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 628636 T1684 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 628637 T1684 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 628637 T1684 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1/'
   [junit4]   2> 628637 T1684 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1/lib/classes/' to classloader
   [junit4]   2> 628638 T1684 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1/lib/README' to classloader
   [junit4]   2> 628666 T1684 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 628704 T1684 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 628705 T1684 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 628711 T1684 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 628977 T1684 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 628979 T1684 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 628981 T1684 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 628984 T1684 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 629004 T1684 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 629004 T1684 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/
   [junit4]   2> 629005 T1684 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@121387e
   [junit4]   2> 629005 T1684 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2
   [junit4]   2> 629005 T1684 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index/
   [junit4]   2> 629006 T1684 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 629006 T1684 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index
   [junit4]   2> 629006 T1684 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=18.9658203125, floorSegmentMB=2.013671875, forceMergeDeletesPctAllowed=28.875788144137676, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3897341227826966
   [junit4]   2> 629009 T1684 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 629009 T1684 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 629011 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 629013 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 629013 T1684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 629013 T1684 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 629013 T1684 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 629014 T1684 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 629014 T1684 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 629014 T1684 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 629015 T1684 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 629015 T1684 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 629015 T1684 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 629017 T1684 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 629019 T1684 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 629020 T1684 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 629020 T1684 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1000120485, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 629021 T1684 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 629021 T1684 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 629021 T1684 oass.SolrIndexSearcher.<init> Opening Searcher@3900b4[collection1] main
   [junit4]   2> 629024 T1685 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3900b4[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 629025 T1684 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 629029 T1619 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 629030 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 629029 T1688 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:36580 collection:collection1 shard:shard1
   [junit4]   2> 629033 T1688 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 629038 T1688 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 629039 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629039 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629040 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629040 T1688 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 629040 T1688 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C594 name=collection1 org.apache.solr.core.SolrCore@180ad17 url=https://127.0.0.1:36580/collection1 node=127.0.0.1:36580_ C594_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:36580, core=collection1, node_name=127.0.0.1:36580_}
   [junit4]   2> 629040 T1688 C594 P36580 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:36580/collection1/
   [junit4]   2> 629040 T1688 C594 P36580 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 629041 T1688 C594 P36580 oasc.SyncStrategy.syncToMe https://127.0.0.1:36580/collection1/ has no replicas
   [junit4]   2> 629041 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 629041 T1688 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:36580/collection1/ shard1
   [junit4]   2> 629041 T1688 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 629043 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629049 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629053 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629104 T1619 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 629105 T1619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 629107 T1619 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 629108 T1619 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:45441
   [junit4]   2> 629110 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 629110 T1619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 629110 T1619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629
   [junit4]   2> 629111 T1619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/'
   [junit4]   2> 629121 T1619 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/solr.xml
   [junit4]   2> 629153 T1619 oasc.CoreContainer.<init> New CoreContainer 7145772
   [junit4]   2> 629153 T1619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/]
   [junit4]   2> 629154 T1619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 629154 T1619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 629155 T1619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 629155 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 629155 T1619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 629155 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 629156 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 629156 T1619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 629156 T1619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 629157 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629157 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629156 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629156 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629160 T1619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 629161 T1619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 629161 T1619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 629161 T1619 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52473/solr
   [junit4]   2> 629162 T1619 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 629162 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 629164 T1699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f11928 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 629164 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 629166 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 629167 T1701 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12fff98 name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 629168 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 629172 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 629201 T1688 oasc.ZkController.register We are https://127.0.0.1:36580/collection1/ and leader is https://127.0.0.1:36580/collection1/
   [junit4]   2> 629202 T1688 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:36580
   [junit4]   2> 629202 T1688 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 629202 T1688 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 629202 T1688 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 629203 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629204 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629204 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629204 T1688 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 629205 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 629206 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36580",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36580_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 629208 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629310 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629310 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629310 T1701 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629310 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629310 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 630174 T1619 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45441_
   [junit4]   2> 630176 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45441_
   [junit4]   2> 630178 T1652 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630178 T1683 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630178 T1643 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630178 T1701 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630178 T1665 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630184 T1702 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 630184 T1702 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 630185 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 630185 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 630186 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 630185 T1702 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 630187 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 630188 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45441",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45441_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 630188 T1644 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 630188 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 630192 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 630294 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 630294 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 630294 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 630294 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 630295 T1701 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 631186 T1702 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 631186 T1702 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1
   [junit4]   2> 631187 T1702 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 631187 T1702 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 631187 T1702 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 631189 T1702 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 631189 T1702 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1/'
   [junit4]   2> 631189 T1702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1/lib/classes/' to classloader
   [junit4]   2> 631189 T1702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1/lib/README' to classloader
   [junit4]   2> 631218 T1702 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 631251 T1702 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 631252 T1702 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 631257 T1702 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 631535 T1702 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 631538 T1702 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 631539 T1702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 631542 T1702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 631563 T1702 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 631563 T1702 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/
   [junit4]   2> 631563 T1702 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@121387e
   [junit4]   2> 631564 T1702 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3
   [junit4]   2> 631564 T1702 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index/
   [junit4]   2> 631565 T1702 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 631565 T1702 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index
   [junit4]   2> 631566 T1702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=18.9658203125, floorSegmentMB=2.013671875, forceMergeDeletesPctAllowed=28.875788144137676, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3897341227826966
   [junit4]   2> 631568 T1702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 631568 T1702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 631570 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 631570 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 631571 T1702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 631571 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 631571 T1702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 631571 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 631572 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 631573 T1702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 631573 T1702 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 631574 T1702 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 631574 T1702 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 631574 T1702 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 631575 T1702 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 631575 T1702 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 631575 T1702 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 631576 T1702 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 631578 T1702 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 631580 T1702 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 631580 T1702 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 631581 T1702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1000120485, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 631582 T1702 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 631582 T1702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 631589 T1702 oass.SolrIndexSearcher.<init> Opening Searcher@37b87c[collection1] main
   [junit4]   2> 631591 T1703 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37b87c[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 631593 T1702 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 631594 T1619 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 631594 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 631594 T1706 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:45441 collection:collection1 shard:shard2
   [junit4]   2> 631599 T1706 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 631603 T1706 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 631605 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 631605 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 631605 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 631605 T1706 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 631606 T1706 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C595 name=collection1 org.apache.solr.core.Solr

[...truncated too long message...]

olExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  79) Thread[id=2089, name=searcherExecutor-1575-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  80) Thread[id=2076, name=qtp32917153-2076 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  81) Thread[id=2028, name=qtp7406006-2028 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  82) Thread[id=2092, name=coreZkRegister-1571-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 7848092 T2060 oaz.ClientCnxn$SendThread.run WARN Session 0x14444acc2ab0029 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene46: {rnd_b=PostingsFormat(name=NestedPulsing), id=PostingsFormat(name=NestedPulsing), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_t=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=pt, timezone=Asia/Kuching
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=97,free=213773560,total=485228544
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsLockFactoryTest, SuggesterWFSTTest, TestFastOutputStream, TestCollationFieldDocValues, TestRTGBase, SpellCheckCollatorTest, DefaultValueUpdateProcessorTest, AssignTest, UniqFieldsUpdateProcessorFactoryTest, TestXIncludeConfig, XmlUpdateRequestHandlerTest, BadCopyFieldTest, TestUniqueKeyFieldResource, ConnectionManagerTest, HdfsSyncSliceTest, TestShardHandlerFactory, TestAnalyzedSuggestions, SpellingQueryConverterTest, TestReloadAndDeleteDocs, TestMultiCoreConfBootstrap, TestDefaultSimilarityFactory, IndexBasedSpellCheckerTest, TestFieldResource, TestDFRSimilarityFactory, TestClassNameShortening, TestRecovery, TestSimpleQParserPlugin, TestStressLucene, HdfsUnloadDistributedZkTest, FileUtilsTest, TestLazyCores, SortByFunctionTest, HighlighterTest, BadIndexSchemaTest, CachingDirectoryFactoryTest, OutputWriterTest, CursorPagingTest, HardAutoCommitTest, IndexSchemaRuntimeFieldTest, TestFieldTypeResource, SolrPluginUtilsTest, NotRequiredUniqueKeyTest, TimeZoneUtilsTest, PreAnalyzedUpdateProcessorTest, TestSchemaResource, TestRandomMergePolicy, TestCodecSupport, TestQuerySenderNoQuery, TestSolrXmlPersistence, DirectSolrSpellCheckerTest, DateFieldTest, TestStressRecovery, OpenCloseCoreStressTest, HdfsRecoveryZkTest, HighlighterConfigTest, TestPartialUpdateDeduplication, TestOmitPositions, TestSolrQueryParserDefaultOperatorResource, TestDistributedMissingSort, SuggesterTSTTest, AlternateDirectoryTest, PreAnalyzedFieldTest, TestWordDelimiterFilterFactory, BasicDistributedZk2Test, CoreContainerCoreInitFailuresTest, MultiTermTest, PrimUtilsTest, TestCursorMarkWithoutUniqueKey, RequiredFieldsTest, SpellCheckComponentTest, TestAddFieldRealTimeGet, TestReload, LeaderElectionIntegrationTest, TestImplicitCoreProperties, TestComponentsName, NumericFieldsTest, TestCSVResponseWriter, ZkSolrClientTest, TestRecoveryHdfs, CurrencyFieldOpenExchangeTest, TestCloudManagedSchema, OpenExchangeRatesOrgProviderTest, SolrCoreCheckLockOnStartupTest, TestCollapseQParserPlugin, TestSolrXml, BasicZkTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=B275C9688947A547 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pt -Dtests.timezone=Asia/Kuching -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B275C9688947A547]:0)
   [junit4] Completed on J1 in 7226.10s, 1 test, 2 errors <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 362 suites, 1563 tests, 1 suite-level error, 1 error, 35 ignored (7 assumptions)

Total time: 155 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b04) - Build # 9407 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9407/
Java: 64bit/jdk1.7.0_60-ea-b04 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([8D2083A9F9731DF1]:0)


REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([8D2083A9F9731DF1]:0)




Build Log:
[...truncated 11017 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> 15092 T59 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 15092 T59 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 15100 T59 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1392708603291
   [junit4]   2> 15103 T59 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 15105 T60 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 15205 T59 oasc.ZkTestServer.run start zk server on port:38204
   [junit4]   2> 15255 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 15275 T61 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 16876 T66 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fd36711 name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 16877 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 16879 T59 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 16909 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 16911 T68 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3118a325 name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 16911 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 16916 T59 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 16921 T59 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 16927 T59 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 16931 T59 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 16936 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 16936 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 16943 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 16944 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 16949 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 16949 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 16954 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 16955 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 16960 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 16960 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 16965 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 16965 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 16969 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 16969 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 16973 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 16973 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 16977 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 16977 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 16981 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 16981 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 16986 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 16986 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 17177 T59 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 17250 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35581
   [junit4]   2> 17277 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 17278 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 17278 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183
   [junit4]   2> 17278 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/'
   [junit4]   2> 17289 T59 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/solr.xml
   [junit4]   2> 17319 T59 oasc.CoreContainer.<init> New CoreContainer 1955836476
   [junit4]   2> 17320 T59 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/]
   [junit4]   2> 17329 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 17329 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 17330 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 17330 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 17330 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 17331 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 17331 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 17331 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 17332 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 17335 T59 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 17335 T59 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 17335 T59 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 17336 T59 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38204/solr
   [junit4]   2> 17343 T59 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 17344 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 17348 T80 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d6271bb name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 17349 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 17356 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 17360 T82 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cb88b36 name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 17360 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 17368 T59 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 17382 T59 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 17396 T59 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 17402 T59 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35581_
   [junit4]   2> 17407 T59 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35581_
   [junit4]   2> 17418 T59 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 17425 T59 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 17446 T59 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:35581_
   [junit4]   2> 17447 T59 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 17454 T59 oasc.Overseer.start Overseer (id=91272551030849539-127.0.0.1:35581_-n_0000000000) starting
   [junit4]   2> 17468 T59 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 17483 T84 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 17484 T59 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 17491 T59 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 17493 T84 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 17498 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 17513 T83 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 17522 T85 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 17523 T85 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 17530 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 17530 T85 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 17536 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 17539 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35581",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35581_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 17540 T83 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 17553 T83 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 17561 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 17569 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 18531 T85 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 18532 T85 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1
   [junit4]   2> 18532 T85 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 18533 T85 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 18534 T85 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 18535 T85 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 18536 T85 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1/'
   [junit4]   2> 18537 T85 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1/lib/classes/' to classloader
   [junit4]   2> 18537 T85 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1/lib/README' to classloader
   [junit4]   2> 18594 T85 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 18622 T85 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 18624 T85 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 18636 T85 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 18895 T85 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 18898 T85 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 18900 T85 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 18906 T85 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 18957 T85 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 18963 T85 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/
   [junit4]   2> 18963 T85 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f3f6de1
   [junit4]   2> 18967 T85 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data
   [junit4]   2> 18967 T85 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index/
   [junit4]   2> 18967 T85 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 18968 T85 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index
   [junit4]   2> 18968 T85 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8205642743276823]
   [junit4]   2> 18971 T85 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 18972 T85 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 18979 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 18980 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 18981 T85 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 18981 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 18981 T85 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 18982 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 18982 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 18983 T85 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 18984 T85 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 18986 T85 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 18987 T85 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 18987 T85 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 18988 T85 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 18988 T85 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 18989 T85 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 18990 T85 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 18996 T85 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 18998 T85 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 18998 T85 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 19000 T85 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.26835472916702285]
   [junit4]   2> 19002 T85 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 19002 T85 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 19002 T85 oass.SolrIndexSearcher.<init> Opening Searcher@7c08f261[collection1] main
   [junit4]   2> 19010 T86 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c08f261[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 19015 T85 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 19016 T59 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 19016 T89 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35581 collection:control_collection shard:shard1
   [junit4]   2> 19016 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 19023 T89 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 19034 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19036 T89 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 19038 T91 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2247c9cd name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19038 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19041 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19041 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 19042 T89 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 19043 T89 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 19044 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C8 name=collection1 org.apache.solr.core.SolrCore@64de9461 url=http://127.0.0.1:35581/collection1 node=127.0.0.1:35581_ C8_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:35581, core=collection1, node_name=127.0.0.1:35581_}
   [junit4]   2> 19044 T89 C8 P35581 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35581/collection1/
   [junit4]   2> 19045 T89 C8 P35581 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 19045 T89 C8 P35581 oasc.SyncStrategy.syncToMe http://127.0.0.1:35581/collection1/ has no replicas
   [junit4]   2> 19046 T89 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35581/collection1/ shard1
   [junit4]   2> 19047 T89 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 19048 T59 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 19054 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19062 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19067 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19069 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 19069 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 19112 T89 oasc.ZkController.register We are http://127.0.0.1:35581/collection1/ and leader is http://127.0.0.1:35581/collection1/
   [junit4]   2> 19113 T89 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35581
   [junit4]   2> 19113 T89 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 19113 T89 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 19114 T89 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 19115 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19116 T89 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 19118 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 19119 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35581",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35581_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 19124 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19151 T59 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 19152 T59 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 19154 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52491
   [junit4]   2> 19155 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 19156 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19156 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239
   [junit4]   2> 19157 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/'
   [junit4]   2> 19169 T59 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/solr.xml
   [junit4]   2> 19198 T59 oasc.CoreContainer.<init> New CoreContainer 135178557
   [junit4]   2> 19198 T59 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/]
   [junit4]   2> 19199 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 19199 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 19199 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 19200 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 19200 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 19200 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 19201 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 19201 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 19201 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 19204 T59 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 19204 T59 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 19205 T59 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 19205 T59 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38204/solr
   [junit4]   2> 19205 T59 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 19206 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19209 T102 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a88e7ec name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19210 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19215 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19218 T104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5443de84 name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19218 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19228 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 19228 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 19232 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 20241 T59 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52491_
   [junit4]   2> 20243 T59 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52491_
   [junit4]   2> 20247 T91 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 20247 T104 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 20247 T82 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 20256 T105 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 20256 T105 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 20258 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20258 T105 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 20258 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20259 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20261 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 20262 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52491",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52491_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 20262 T83 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 20262 T83 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 20267 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20371 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 20371 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 20371 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 21258 T105 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 21259 T105 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1
   [junit4]   2> 21259 T105 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 21260 T105 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 21260 T105 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 21262 T105 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 21262 T105 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1/'
   [junit4]   2> 21263 T105 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1/lib/classes/' to classloader
   [junit4]   2> 21263 T105 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1/lib/README' to classloader
   [junit4]   2> 21309 T105 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 21335 T105 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 21338 T105 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 21347 T105 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 21577 T105 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 21579 T105 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 21581 T105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 21585 T105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 21633 T105 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 21633 T105 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/
   [junit4]   2> 21634 T105 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f3f6de1
   [junit4]   2> 21636 T105 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1
   [junit4]   2> 21636 T105 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index/
   [junit4]   2> 21636 T105 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 21637 T105 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index
   [junit4]   2> 21637 T105 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8205642743276823]
   [junit4]   2> 21639 T105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 21640 T105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 21650 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 21651 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 21651 T105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 21652 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 21652 T105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 21652 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 21653 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 21654 T105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 21654 T105 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 21655 T105 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 21657 T105 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 21657 T105 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 21658 T105 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 21660 T105 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 21660 T105 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 21661 T105 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 21663 T105 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21665 T105 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 21665 T105 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 21667 T105 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.26835472916702285]
   [junit4]   2> 21668 T105 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 21668 T105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 21669 T105 oass.SolrIndexSearcher.<init> Opening Searcher@33157362[collection1] main
   [junit4]   2> 21682 T106 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33157362[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 21691 T105 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 21692 T59 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 21693 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 21692 T109 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52491 collection:collection1 shard:shard2
   [junit4]   2> 21696 T109 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 21710 T109 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 21715 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21715 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21716 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21716 T109 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 21716 T109 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@1eb42164 url=http://127.0.0.1:52491/collection1 node=127.0.0.1:52491_ C9_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52491, core=collection1, node_name=127.0.0.1:52491_}
   [junit4]   2> 21717 T109 C9 P52491 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52491/collection1/
   [junit4]   2> 21718 T109 C9 P52491 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 21718 T109 C9 P52491 oasc.SyncStrategy.syncToMe http://127.0.0.1:52491/collection1/ has no replicas
   [junit4]   2> 21719 T109 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52491/collection1/ shard2
   [junit4]   2> 21719 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21719 T109 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 21725 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21728 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21733 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21782 T59 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 21783 T59 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 21785 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48646
   [junit4]   2> 21786 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 21786 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 21787 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884
   [junit4]   2> 21787 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/'
   [junit4]   2> 21806 T59 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/solr.xml
   [junit4]   2> 21835 T59 oasc.CoreContainer.<init> New CoreContainer 458465728
   [junit4]   2> 21836 T59 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/]
   [junit4]   2> 21837 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 21838 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 21838 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 21838 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 21838 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 21839 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 21840 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 21840 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 21841 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 21841 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 21841 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 21842 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 21845 T59 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 21846 T59 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 21846 T59 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 21846 T59 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38204/solr
   [junit4]   2> 21847 T59 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 21848 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21849 T120 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50f752ee name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21850 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21854 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21856 T122 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39978617 name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21857 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21868 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 21880 T109 oasc.ZkController.register We are http://127.0.0.1:52491/collection1/ and leader is http://127.0.0.1:52491/collection1/
   [junit4]   2> 21880 T109 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52491
   [junit4]   2> 21880 T109 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 21881 T109 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 21881 T109 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 21884 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21884 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21885 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21885 T109 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21889 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21892 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52491",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52491_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 21899 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22005 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22005 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22005 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22005 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22877 T59 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48646_
   [junit4]   2> 22880 T59 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48646_
   [junit4]   2> 22887 T104 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22887 T82 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22888 T122 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22887 T91 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22903 T123 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 22903 T123 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 22906 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22906 T123 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 22906 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22907 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22912 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 22914 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48646",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48646_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 22915 T83 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 22915 T83 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 22923 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23029 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 23029 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 23030 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 23029 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 23907 T123 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 23908 T123 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1
   [junit4]   2> 23908 T123 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 23917 T123 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 23917 T123 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 23919 T123 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 23919 T123 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1/'
   [junit4]   2> 23920 T123 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1/lib/classes/' to classloader
   [junit4]   2> 23921 T123 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1/lib/README' to classloader
   [junit4]   2> 23962 T123 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 23991 T123 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 23993 T123 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 24001 T123 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 24254 T123 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 24255 T123 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 24257 T123 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 24260 T123 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 24305 T123 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 24306 T123 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/
   [junit4]   2> 24306 T123 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f3f6de1
   [junit4]   2> 24307 T123 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2
   [junit4]   2> 24308 T123 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index/
   [junit4]   2> 24308 T123 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 24309 T123 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index
   [junit4]   2> 24310 T123 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8205642743276823]
   [junit4]   2> 24400 T123 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 24400 T123 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 24408 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 24408 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 24409 T123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 24409 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 24409 T123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 24409 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 24410 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 24410 T123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 24410 T123 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 24412 T123 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 24413 T123 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 24414 T123 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 24414 T123 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 24415 T123 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 24416 T123 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 24417 T123 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 24419 T123 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 24421 T123 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 24421 T123 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 24423 T123 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.26835472916702285]
   [junit4]   2> 24425 T123 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 24425 T123 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 24425 T123 oass.SolrIndexSearcher.<init> Opening Searcher@109a311d[collection1] main
   [junit4]   2> 24436 T124 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@109a311d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 24443 T123 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 24444 T59 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 24445 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 24444 T127 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48646 collection:collection1 shard:shard1
   [junit4]   2> 24452 T127 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 24468 T127 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 24472 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24472 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24472 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24473 T127 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 24473 T127 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C10 name=collection1 org.apache.solr.core.SolrCore@5512520e url=http://127.0.0.1:48646/collection1 node=127.0.0.1:48646_ C10_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:48646, core=collection1, node_name=127.0.0.1:48646_}
   [junit4]   2> 24473 T127 C10 P48646 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48646/collection1/
   [junit4]   2> 24473 T127 C10 P48646 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 24474 T127 C10 P48646 oasc.SyncStrategy.syncToMe http://127.0.0.1:48646/collection1/ has no replicas
   [junit4]   2> 24474 T127 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48646/collection1/ shard1
   [junit4]   2> 24474 T127 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 24477 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 24487 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24490 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24543 T59 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 24544 T59 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 24546 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43362
   [junit4]   2> 24547 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 24547 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 24547 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636
   [junit4]   2> 24548 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/'
   [junit4]   2> 24568 T59 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/solr.xml
   [junit4]   2> 24601 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24601 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24601 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24603 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24618 T59 oasc.CoreContainer.<init> New CoreContainer 1165359000
   [junit4]   2> 24619 T59 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/]
   [junit4]   2> 24620 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 24621 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 24621 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 24622 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 24622 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 24623 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 24623 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 24623 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 24624 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 24628 T59 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 24629 T59 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 24629 T59 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 24630 T59 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38204/solr
   [junit4]   2> 24630 T59 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 24631 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24634 T138 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43d23fc8 name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24635 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24637 T127 oasc.ZkController.register We are http://127.0.0.1:48646/collection1/ and leader is http://127.0.0.1:48646/collection1/
   [junit4]   2> 24638 T61 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443e6cdf00009, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 24638 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24638 T127 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48646
   [junit4]   2> 24639 T127 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 24639 T127 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 24640 T127 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 24640 T140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a1871df name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24641 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24641 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24642 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24642 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24642 T127 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 24645 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 24646 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48646",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48646_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 24648 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 24650 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24753 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24753 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24753 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24753 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24753 T140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 25653 T59 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43362_
   [junit4]   2> 25655 T59 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43362_
   [junit4]   2> 25658 T122 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25658 T91 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25659 T104 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25659 T140 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25659 T82 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25667 T141 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 25668 T141 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 25670 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25670 T141 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 25670 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25671 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25673 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 25674 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43362",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43362_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 25675 T83 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 25675 T83 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 25681 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25784 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 25785 T140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 25785 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 25785 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 25784 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 26671 T141 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 26671 T141 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1
   [junit4]   2> 26672 T141 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 26674 T141 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 26674 T141 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 26677 T141 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 26677 T141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1/'
   [junit4]   2> 26679 T141 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1/lib/classes/' to classloader
   [junit4]   2> 26680 T141 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1/lib/README' to classloader
   [junit4]   2> 26720 T141 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 26746 T141 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 26750 T141 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 26759 T141 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 26996 T141 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 26998 T141 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 26999 T141 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 27003 T141 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 27041 T141 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 27041 T141 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/
   [junit4]   2> 27042 T141 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f3f6de1
   [junit4]   2> 27043 T141 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3
   [junit4]   2> 27043 T141 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index/
   [junit4]   2> 27043 T141 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 27044 T141 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index
   [junit4]   2> 27044 T141 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8205642743276823]
   [junit4]   2> 27046 T141 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 27047 T141 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 27054 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 27055 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 27055 T141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 27055 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 27056 T141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 27056 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 27056 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 27057 T141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 27057 T141 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 27058 T141 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 27059 T141 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 27059 T141 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 27060 T141 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 27061 T141 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 27061 T141 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 27062 T141 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 27064 T141 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 27066 T141 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 27067 T141 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 27068 T141 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.26835472916702285]
   [junit4]   2> 27069 T141 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 27070 T141 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 27070 T141 oass.SolrIndexSearcher.<init> Opening Searcher@69eb0ef1[collection1] main
   [junit4]   2> 27077 T142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69eb0ef1[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 27082 T141 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 27082 T59 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 27083 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 27082 T145 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43362 collection:collection1 shard:shard3
   [junit4]   2> 27085 T145 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 27095 T145 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 27098 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27098 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27099 T145 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 27099 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27099 T145 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@19080dcc url=http://127.0.0.1:43362/collection1 node=127.0.0.1:43362_ C11_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:43362, core=collection1, node_name=127.0.0.1:43362_}
   [junit4]   2> 27100 T145 C11 P43362 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43362/collection1/
   [junit4]   2> 27100 T145 C11 P43362 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 27100 T1

[...truncated too long message...]

   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  68) Thread[id=235, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[8D2083A9F9731DF1]-SendThread(localhost.localdomain:38204), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 	  69) Thread[id=286, name=qtp2002619729-286 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  70) Thread[id=227, name=qtp485339482-227 Acceptor0 SelectChannelConnector@127.0.0.1:52063, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  71) Thread[id=535, name=coreZkRegister-237-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  72) Thread[id=288, name=qtp2002619729-288 Acceptor1 SelectChannelConnector@127.0.0.1:55023, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  73) Thread[id=547, name=qtp859089295-547 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  74) Thread[id=342, name=commitScheduler-134-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  75) Thread[id=284, name=HashSessionScavenger-11, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {id=PostingsFormat(name=Direct), rnd_b=PostingsFormat(name=Direct), _version_=PostingsFormat(name=Asserting), a_t=PostingsFormat(name=Asserting), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings)}, docValues:{}, sim=DefaultSimilarity, locale=it, timezone=Africa/Brazzaville
   [junit4]   2> NOTE: Linux 3.8.0-35-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=87,free=35082112,total=126418944
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollationFieldDocValues, DirectSolrSpellCheckerTest, DocValuesMultiTest, SolrPluginUtilsTest, SolrCoreCheckLockOnStartupTest, TestPartialUpdateDeduplication, ChangedSchemaMergeTest, TestQuerySenderNoQuery, CursorMarkTest, SpellingQueryConverterTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=8D2083A9F9731DF1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it -Dtests.timezone=Africa/Brazzaville -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8D2083A9F9731DF1]:0)
   [junit4] Completed on J1 in 7225.69s, 1 test, 2 errors <<< FAILURES!

[...truncated 251 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 362 suites, 1563 tests, 1 suite-level error, 1 error, 304 ignored (5 assumptions)

Total time: 143 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b04 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b04) - Build # 9406 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9406/
Java: 64bit/jdk1.7.0_60-ea-b04 -XX:+UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.testDistribSearch

Error Message:
Expected the partialResults header to be null

Stack Trace:
java.lang.AssertionError: Expected the partialResults header to be null
	at __randomizedtesting.SeedInfo.seed([80C27A6F03501905:124F477740F7939]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.TestDistributedSearch.validateControlData(TestDistributedSearch.java:525)
	at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:445)
	at org.apache.solr.TestDistributedSearch.doTest(TestDistributedSearch.java:379)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:44025/fc/l/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:44025/fc/l/collection1
	at __randomizedtesting.SeedInfo.seed([80C27A6F03501905:124F477740F7939]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:503)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.index_specific(AbstractFullDistribZkTestBase.java:698)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingWithSuss(FullSolrCloudDistribCmdsTest.java:443)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:134)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
	... 48 more




Build Log:
[...truncated 10529 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> 1057710 T4534 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1057710 T4534 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /fc/l
   [junit4]   2> 1057713 T4534 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-FullSolrCloudDistribCmdsTest-1392702755331
   [junit4]   2> 1057714 T4534 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1057714 T4535 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1057814 T4534 oasc.ZkTestServer.run start zk server on port:39855
   [junit4]   2> 1057815 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1057900 T4541 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@215494dc name:ZooKeeperConnection Watcher:127.0.0.1:39855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1057900 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1057901 T4534 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1057905 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1057906 T4543 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47d8f373 name:ZooKeeperConnection Watcher:127.0.0.1:39855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1057907 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1057907 T4534 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1057910 T4534 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1057912 T4534 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1057914 T4534 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1057916 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1057917 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1057922 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1057923 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1057925 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1057925 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1057927 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1057927 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1057929 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1057929 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1057932 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1057932 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1057934 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1057934 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1057936 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1057937 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1057938 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1057939 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1057941 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1057941 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1057943 T4534 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1057943 T4534 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1057946 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1057947 T4545 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2de2c84e name:ZooKeeperConnection Watcher:127.0.0.1:39855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1057947 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1058020 T4534 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1058023 T4534 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1058024 T4534 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:44025
   [junit4]   2> 1058027 T4534 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1058027 T4534 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1058028 T4534 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1392702755568
   [junit4]   2> 1058028 T4534 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1392702755568/'
   [junit4]   2> 1058037 T4534 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1392702755568/solr.xml
   [junit4]   2> 1058069 T4534 oasc.CoreContainer.<init> New CoreContainer 253411485
   [junit4]   2> 1058069 T4534 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1392702755568/]
   [junit4]   2> 1058070 T4534 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1058070 T4534 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1058070 T4534 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1058070 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1058071 T4534 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1058071 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1058071 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1058071 T4534 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1058071 T4534 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1058075 T4534 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1058075 T4534 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1058075 T4534 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1058076 T4534 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39855/solr
   [junit4]   2> 1058076 T4534 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1058077 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1058079 T4556 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@655f1ef2 name:ZooKeeperConnection Watcher:127.0.0.1:39855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1058080 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1058082 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1058083 T4558 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f260823 name:ZooKeeperConnection Watcher:127.0.0.1:39855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1058083 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1058085 T4534 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1058087 T4534 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1058090 T4534 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1058091 T4534 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44025_fc%2Fl
   [junit4]   2> 1058092 T4534 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44025_fc%2Fl
   [junit4]   2> 1058094 T4534 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1058095 T4534 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1058097 T4534 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44025_fc%2Fl
   [junit4]   2> 1058097 T4534 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1058098 T4534 oasc.Overseer.start Overseer (id=91272167773503492-127.0.0.1:44025_fc%2Fl-n_0000000000) starting
   [junit4]   2> 1058100 T4534 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1058102 T4560 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1058102 T4534 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1058103 T4560 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1058105 T4534 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1058106 T4534 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1058107 T4559 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1058110 T4561 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1058110 T4561 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1058111 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1058111 T4561 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1058112 T4559 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1058113 T4559 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44025/fc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44025_fc%2Fl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1058113 T4559 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1058113 T4559 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1058115 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1058115 T4558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1059112 T4561 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1059112 T4561 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1392702755568/collection1
   [junit4]   2> 1059112 T4561 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1059113 T4561 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1059113 T4561 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1059114 T4561 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1059114 T4561 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1392702755568/collection1/'
   [junit4]   2> 1059115 T4561 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1392702755568/collection1/lib/classes/' to classloader
   [junit4]   2> 1059115 T4561 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1392702755568/collection1/lib/README' to classloader
   [junit4]   2> 1059150 T4561 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1059172 T4561 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1059173 T4561 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1059179 T4561 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1059378 T4561 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1059379 T4561 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1059380 T4561 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1059383 T4561 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1059400 T4561 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1059401 T4561 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1392702755568/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/control/data/
   [junit4]   2> 1059401 T4561 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5b356c94
   [junit4]   2> 1059401 T4561 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/control/data
   [junit4]   2> 1059401 T4561 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/control/data/index/
   [junit4]   2> 1059401 T4561 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1059402 T4561 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/control/data/index
   [junit4]   2> 1059402 T4561 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1716305150, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1059403 T4561 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@19a15dfa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@26006eb6),segFN=segments_1,generation=1}
   [junit4]   2> 1059403 T4561 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1059405 T4561 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1059405 T4561 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1059405 T4561 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1059405 T4561 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1059406 T4561 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1059406 T4561 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1059406 T4561 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1059406 T4561 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1059406 T4561 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1059407 T4561 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1059407 T4561 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1059407 T4561 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1059407 T4561 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1059407 T4561 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1059408 T4561 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1059408 T4561 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1059410 T4561 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1059412 T4561 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1059412 T4561 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1059412 T4561 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=8, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1059413 T4561 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@19a15dfa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@26006eb6),segFN=segments_1,generation=1}
   [junit4]   2> 1059413 T4561 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1059413 T4561 oass.SolrIndexSearcher.<init> Opening Searcher@1d967441[collection1] main
   [junit4]   2> 1059415 T4562 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d967441[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1059415 T4561 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1059416 T4534 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1059416 T4534 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1059416 T4565 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:44025/fc/l collection:control_collection shard:shard1
   [junit4]   2> 1059417 T4565 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1059419 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1059421 T4567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72a50ec7 name:ZooKeeperConnection Watcher:127.0.0.1:39855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1059421 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1059421 T4565 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1059422 T4534 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1059423 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059423 T4565 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1059423 T4565 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1059424 T4534 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1059423 T4559 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C1302 name=collection1 org.apache.solr.core.SolrCore@1b155bf1 url=https://127.0.0.1:44025/fc/l/collection1 node=127.0.0.1:44025_fc%2Fl C1302_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:44025/fc/l, core=collection1, node_name=127.0.0.1:44025_fc%2Fl}
   [junit4]   2> 1059424 T4565 C1302 P44025 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:44025/fc/l/collection1/
   [junit4]   2> 1059424 T4565 C1302 P44025 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1059424 T4565 C1302 P44025 oasc.SyncStrategy.syncToMe https://127.0.0.1:44025/fc/l/collection1/ has no replicas
   [junit4]   2> 1059425 T4565 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:44025/fc/l/collection1/ shard1
   [junit4]   2> 1059425 T4565 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1059426 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059429 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059431 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059509 T4534 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1059510 T4534 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1059512 T4534 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1059513 T4534 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53845
   [junit4]   2> 1059515 T4534 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1059516 T4534 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1059516 T4534 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1392702757042
   [junit4]   2> 1059516 T4534 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1392702757042/'
   [junit4]   2> 1059528 T4534 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1392702757042/solr.xml
   [junit4]   2> 1059534 T4558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1059534 T4567 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1059553 T4534 oasc.CoreContainer.<init> New CoreContainer 1536795584
   [junit4]   2> 1059553 T4534 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1392702757042/]
   [junit4]   2> 1059554 T4534 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1059554 T4534 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1059554 T4534 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1059555 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1059555 T4534 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1059555 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1059555 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1059556 T4534 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1059556 T4534 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1059559 T4534 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1059560 T4534 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1059560 T4534 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1059560 T4534 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39855/solr
   [junit4]   2> 1059560 T4534 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1059561 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1059562 T4578 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60a09aa1 name:ZooKeeperConnection Watcher:127.0.0.1:39855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1059563 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1059564 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1059565 T4580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64e2839a name:ZooKeeperConnection Watcher:127.0.0.1:39855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1059566 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1059568 T4534 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1059580 T4565 oasc.ZkController.register We are https://127.0.0.1:44025/fc/l/collection1/ and leader is https://127.0.0.1:44025/fc/l/collection1/
   [junit4]   2> 1059580 T4565 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:44025/fc/l
   [junit4]   2> 1059580 T4565 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1059580 T4565 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1059580 T4565 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1059581 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059581 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059581 T4565 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1059581 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059582 T4559 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1059583 T4559 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44025/fc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44025_fc%2Fl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1059584 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059686 T4580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1059686 T4558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1059686 T4567 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1060570 T4534 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53845_fc%2Fl
   [junit4]   2> 1060571 T4534 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53845_fc%2Fl
   [junit4]   2> 1060573 T4580 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1060573 T4558 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1060573 T4567 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1060578 T4581 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1060578 T4581 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1060579 T4581 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1060579 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1060579 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1060580 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1060580 T4559 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1060581 T4559 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53845/fc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53845_fc%2Fl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1060581 T4559 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 1060581 T4559 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1060583 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1060686 T4580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1060686 T4567 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1060686 T4558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1061579 T4581 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1061580 T4581 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1392702757042/collection1
   [junit4]   2> 1061580 T4581 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1061580 T4581 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1061581 T4581 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1061581 T4581 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1061582 T4581 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1392702757042/collection1/'
   [junit4]   2> 1061582 T4581 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1392702757042/collection1/lib/classes/' to classloader
   [junit4]   2> 1061583 T4581 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1392702757042/collection1/lib/README' to classloader
   [junit4]   2> 1061616 T4581 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1061637 T4581 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1061639 T4581 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1061644 T4581 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1061827 T4581 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1061828 T4581 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1061829 T4581 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1061832 T4581 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1061848 T4581 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1061848 T4581 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1392702757042/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty1/
   [junit4]   2> 1061848 T4581 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5b356c94
   [junit4]   2> 1061849 T4581 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty1
   [junit4]   2> 1061849 T4581 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty1/index/
   [junit4]   2> 1061849 T4581 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1061849 T4581 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty1/index
   [junit4]   2> 1061850 T4581 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1716305150, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1061851 T4581 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@17499ea lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b751d27),segFN=segments_1,generation=1}
   [junit4]   2> 1061851 T4581 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1061853 T4581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1061853 T4581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1061854 T4581 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1061854 T4581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1061854 T4581 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1061854 T4581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1061854 T4581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1061855 T4581 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1061855 T4581 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1061855 T4581 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1061855 T4581 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1061856 T4581 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1061856 T4581 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1061856 T4581 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1061856 T4581 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1061857 T4581 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1061859 T4581 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1061860 T4581 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1061861 T4581 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1061861 T4581 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=8, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1061862 T4581 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@17499ea lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b751d27),segFN=segments_1,generation=1}
   [junit4]   2> 1061862 T4581 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1061862 T4581 oass.SolrIndexSearcher.<init> Opening Searcher@31a86afb[collection1] main
   [junit4]   2> 1061864 T4582 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31a86afb[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1061865 T4581 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1061866 T4534 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1061866 T4534 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1061866 T4585 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:53845/fc/l collection:collection1 shard:shard1
   [junit4]   2> 1061871 T4585 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1061875 T4585 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1061876 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1061876 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1061876 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1061877 T4585 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1061877 T4585 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1303 name=collection1 org.apache.solr.core.SolrCore@1072c31f url=https://127.0.0.1:53845/fc/l/collection1 node=127.0.0.1:53845_fc%2Fl C1303_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:53845/fc/l, core=collection1, node_name=127.0.0.1:53845_fc%2Fl}
   [junit4]   2> 1061877 T4585 C1303 P53845 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:53845/fc/l/collection1/
   [junit4]   2> 1061877 T4585 C1303 P53845 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1061877 T4559 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1061878 T4585 C1303 P53845 oasc.SyncStrategy.syncToMe https://127.0.0.1:53845/fc/l/collection1/ has no replicas
   [junit4]   2> 1061878 T4585 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:53845/fc/l/collection1/ shard1
   [junit4]   2> 1061878 T4585 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1061880 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1061882 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1061884 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1061933 T4534 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1061934 T4534 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1061937 T4534 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1061937 T4534 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35056
   [junit4]   2> 1061939 T4534 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1061940 T4534 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1061940 T4534 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1392702759485
   [junit4]   2> 1061940 T4534 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1392702759485/'
   [junit4]   2> 1061950 T4534 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1392702759485/solr.xml
   [junit4]   2> 1061971 T4534 oasc.CoreContainer.<init> New CoreContainer 1983506034
   [junit4]   2> 1061971 T4534 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1392702759485/]
   [junit4]   2> 1061972 T4534 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1061972 T4534 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1061972 T4534 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1061973 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1061973 T4534 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1061973 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1061973 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1061973 T4534 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1061973 T4534 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1061977 T4534 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1061978 T4534 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1061978 T4534 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1061979 T4534 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39855/solr
   [junit4]   2> 1061979 T4534 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1061980 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1061982 T4596 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ff131fa name:ZooKeeperConnection Watcher:127.0.0.1:39855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1061982 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1061984 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1061986 T4598 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b6f51c9 name:ZooKeeperConnection Watcher:127.0.0.1:39855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1061986 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1061987 T4567 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1061987 T4558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1061987 T4580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1061990 T4534 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1062033 T4585 oasc.ZkController.register We are https://127.0.0.1:53845/fc/l/collection1/ and leader is https://127.0.0.1:53845/fc/l/collection1/
   [junit4]   2> 1062033 T4585 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:53845/fc/l
   [junit4]   2> 1062033 T4585 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1062033 T4585 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1062033 T4585 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1062034 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1062034 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1062035 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1062038 T4585 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1062039 T4559 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1062040 T4559 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53845/fc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53845_fc%2Fl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1062041 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1062143 T4558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1062144 T4598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1062143 T4567 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1062143 T4580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1062993 T4534 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35056_fc%2Fl
   [junit4]   2> 1062994 T4534 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35056_fc%2Fl
   [junit4]   2> 1062995 T4567 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1062995 T4558 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1062996 T4580 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1062996 T4598 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1063000 T4599 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1063000 T4599 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1063001 T4599 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1063001 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1063001 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1063001 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1063002 T4559 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1063002 T4559 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35056/fc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35056_fc%2Fl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1063002 T4559 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1063002 T4559 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1063004 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1063107 T4567 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1063107 T4598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1063107 T4580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1063107 T4558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1064001 T4599 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1064001 T4599 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1392702759485/collection1
   [junit4]   2> 1064001 T4599 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1064002 T4599 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1064002 T4599 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1064003 T4599 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1064003 T4599 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1392702759485/collection1/'
   [junit4]   2> 1064003 T4599 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1392702759485/collection1/lib/classes/' to classloader
   [junit4]   2> 1064004 T4599 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1392702759485/collection1/lib/README' to classloader
   [junit4]   2> 1064047 T4599 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1064078 T4599 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1064079 T4599 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1064085 T4599 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1064320 T4599 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1064323 T4599 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1064324 T4599 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1064328 T4599 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1064354 T4599 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1064355 T4599 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1392702759485/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty2/
   [junit4]   2> 1064355 T4599 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5b356c94
   [junit4]   2> 1064356 T4599 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty2
   [junit4]   2> 1064356 T4599 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty2/index/
   [junit4]   2> 1064356 T4599 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1064357 T4599 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1392702755331/jetty2/index
   [junit4]   2> 1064358 T4599 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1716305150, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1064358 T4599 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@12b04234 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@252e10f6),segFN=segments_1,generation=1}
   [junit4]   2> 1064359 T4599 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1064363 T4599 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1064363 T4599 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1064364 T4599 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1064364 T4599 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1064364 T4599 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1064364 T4599 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1064365 T4599 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1064365 T4599 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1064365 T4599 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1064366 T4599 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1064367 T4599 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1064367 T4599 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1064367 T4599 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1064368 T4599 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1064368 T4599 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1064368 T4599 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1064372 T4599 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1064374 T4599 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1064375 T4599 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1064375 T4599 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=8, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1064376 T4599 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@12b04234 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@252e10f6),segFN=segments_1,generation=1}
   [junit4]   2> 1064376 T4599 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1064376 T4599 oass.SolrIndexSearcher.<init> Opening Searcher@5ccedbed[collection1] main
   [junit4]   2> 1064380 T4600 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5ccedbed[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1064382 T4599 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1064383 T4534 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1064383 T4534 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1064383 T4603 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:35056/fc/l collection:collection1 shard:shard3
   [junit4]   2> 1064385 T4603 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 1064391 T4603 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 1064393 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064393 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064393 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064394 T4603 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1064394 T4603 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1064394 T4559 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C1304 name=collection1 org.apache.solr.core.SolrCore@4d6bab74 url=https://127.0.0.1:35056/fc/l/collection1 node=127.0.0.1:35056_fc%2Fl C1304_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:35056/fc/l, core=collection1, node_name=127.0.0.1:35056_fc%2Fl}
   [junit4]   2> 1064395 T4603 C1304 P35056 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:35056/fc/l/collection1/
   [junit4]   2> 1064395 T4603 C1304 P35056 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1064395 T4603 C1304 P35056 oasc.SyncStrategy.syncToMe https://127.0.0.1:35056/fc/l/collection1/ has no replicas
   [junit4]   2> 1064395 T4603 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:35056/fc/l/collection1/ shard3
   [junit4]   2> 1064396 T4603 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 1064397 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064400 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064403 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064482 T4534 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1064483 T4534 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1064486 T4534 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1064487 T4534 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:46239
   [junit4]   2> 1064489 T4534 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1064489 T4534 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1064490 T4534 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1392702762003
   [junit4]   2> 1064490 T4534 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1392702762003/'
   [junit4]   2> 1064504 T4534 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1392702762003/solr.xml
   [junit4]   2> 1064505 T4558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1064505 T4580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1064505 T4567 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1064506 T4598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1064534 T4534 oasc.CoreContainer.<init> New CoreContainer 1184706657
   [junit4]   2> 1064535 T4534 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1392702762003/]
   [junit4]   2> 1064536 T4534 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1064536 T4534 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1064536 T4534 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1064536 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1064537 T4534 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1064537 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1064537 T4534 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1064537 T4534 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1064537 T4534 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1064541 T4534 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1064541 T4534 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1064541 T4534 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1064541 T4534 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39855/solr
   [junit4]   2> 1064542 T4534 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1064542 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1064544 T4614 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6597d75e name:ZooKeeperConnection Watcher:127.0.0.1:39855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1064545 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1064547 T4534 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1064548 T4616 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e84a813 name:ZooKeeperConnection Watcher:127.0.0.1:39855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1064548 T4534 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1064551 T4534 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1064551 T4603 oasc.ZkController.register We are https://127.0.0.1:35056/fc/l/collection1/ and leader is https://127.0.0.1:35056/fc/l/collection1/
   [junit4]   2> 1064551 T4603 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:35056/fc/l
   [junit4]   2> 1064552 T4603 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1064552 T4603 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1064552 T4603 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1064553 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064553 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064553 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064554 T4603 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1064554 T4559 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1064556 T4559 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:35056/fc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35056_fc%2Fl",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1064558 T4558 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1064659 T4567 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1064660 T4616 oascc.ZkStateReader$2.process 

[...truncated too long message...]

ing directory: ./org.apache.solr.TestDistributedSearch-1392702914292/control/data/index
   [junit4]   2> 1230487 T5443 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.TestDistributedSearch-1392702914292/control/data [CachedDir<<refCount=0;path=./org.apache.solr.TestDistributedSearch-1392702914292/control/data;done=false>>]
   [junit4]   2> 1230487 T5443 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.TestDistributedSearch-1392702914292/control/data
   [junit4]   2> 1230488 T5443 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/a/k,null}
   [junit4]   2> 1230580 T5443 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1773008697
   [junit4]   2> 1230581 T5443 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@28fe43f8
   [junit4]   2> 1230593 T5443 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=3,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=35,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=9530,transaction_logs_total_number=3}
   [junit4]   2> 1230594 T5443 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1230594 T5443 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1230595 T5443 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1230596 T5443 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1230597 T5443 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1230597 T5443 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.TestDistributedSearch-1392702914292/shard0/data [CachedDir<<refCount=0;path=./org.apache.solr.TestDistributedSearch-1392702914292/shard0/data;done=false>>]
   [junit4]   2> 1230597 T5443 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.TestDistributedSearch-1392702914292/shard0/data
   [junit4]   2> 1230598 T5443 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.TestDistributedSearch-1392702914292/shard0/data/index [CachedDir<<refCount=0;path=./org.apache.solr.TestDistributedSearch-1392702914292/shard0/data/index;done=false>>]
   [junit4]   2> 1230598 T5443 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.TestDistributedSearch-1392702914292/shard0/data/index
   [junit4]   2> 1230605 T5443 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/a/k,null}
   [junit4]   2> 1230680 T5443 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2007168616
   [junit4]   2> 1230681 T5443 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5e1690ee
   [junit4]   2> 1230697 T5443 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=3,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=36,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=8897,transaction_logs_total_number=3}
   [junit4]   2> 1230697 T5443 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1230697 T5443 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1230698 T5443 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1230699 T5443 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1230699 T5443 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1230699 T5443 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.TestDistributedSearch-1392702914292/shard1/data [CachedDir<<refCount=0;path=./org.apache.solr.TestDistributedSearch-1392702914292/shard1/data;done=false>>]
   [junit4]   2> 1230700 T5443 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.TestDistributedSearch-1392702914292/shard1/data
   [junit4]   2> 1230700 T5443 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.TestDistributedSearch-1392702914292/shard1/data/index [CachedDir<<refCount=0;path=./org.apache.solr.TestDistributedSearch-1392702914292/shard1/data/index;done=false>>]
   [junit4]   2> 1230700 T5443 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.TestDistributedSearch-1392702914292/shard1/data/index
   [junit4]   2> 1230703 T5443 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/a/k,null}
   [junit4]   2> 1230785 T5443 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDistributedSearch -Dtests.method=testDistribSearch -Dtests.seed=80C27A6F03501905 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=in_ID -Dtests.timezone=America/Moncton -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 14.1s J0 | TestDistributedSearch.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Expected the partialResults header to be null
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([80C27A6F03501905:124F477740F7939]:0)
   [junit4]    > 	at org.apache.solr.TestDistributedSearch.validateControlData(TestDistributedSearch.java:525)
   [junit4]    > 	at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:445)
   [junit4]    > 	at org.apache.solr.TestDistributedSearch.doTest(TestDistributedSearch.java:379)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 1230791 T5443 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=in_ID, timezone=America/Moncton
   [junit4]   2> NOTE: Linux 3.8.0-35-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=238836800,total=430440448
   [junit4]   2> NOTE: All tests run in this JVM: [OverseerCollectionProcessorTest, BasicFunctionalityTest, SliceStateUpdateTest, CustomCollectionTest, LoggingHandlerTest, SolrCoreTest, CacheHeaderTest, TestCoreContainer, TestExtendedDismaxParser, ClusterStateTest, TriLevelCompositeIdRoutingTest, SearchHandlerTest, TestCloudManagedSchemaAddField, UpdateParamsTest, SampleTest, TestSolrIndexConfig, AutoCommitTest, TestFieldCollectionResource, SuggesterTest, TestSolrXMLSerializer, ReturnFieldsTest, FieldMutatingUpdateProcessorTest, TestNoOpRegenerator, TestStandardQParsers, SpatialFilterTest, TestValueSourceCache, TestLFUCache, AliasIntegrationTest, BasicDistributedZkTest, PrimitiveFieldTypeTest, DirectUpdateHandlerOptimizeTest, LeaderElectionTest, LegacyHTMLStripCharFilterTest, ShardSplitTest, SolrInfoMBeanTest, ClusterStateUpdateTest, InfoHandlerTest, QueryElevationComponentTest, TestFaceting, AddBlockUpdateTest, TermsComponentTest, TestOverriddenPrefixQueryForCustomFieldType, AnalysisAfterCoreReloadTest, QueryParsingTest, TestDocSet, RegexBoostProcessorTest, IndexSchemaTest, TestJmxMonitoredMap, TestPostingsSolrHighlighter, TestGroupingSearch, OverseerRolesTest, ShowFileRequestHandlerTest, DocumentBuilderTest, TestMergePolicyConfig, TestPseudoReturnFields, UUIDFieldTest, SimpleFacetsTest, CopyFieldTest, TestRemoteStreaming, SchemaVersionSpecificBehaviorTest, TestSystemIdResolver, ShardRoutingTest, DistributedDebugComponentTest, TestNonNRTOpen, SolrTestCaseJ4Test, TestTrie, TestCustomSort, TermVectorComponentTest, TestElisionMultitermQuery, EnumFieldTest, BlockCacheTest, TestSurroundQueryParser, ExternalFileFieldSortTest, TestRealTimeGet, TestNRTOpen, MBeansHandlerTest, DeleteReplicaTest, TestInitQParser, TestStressVersions, JsonLoaderTest, SOLR749Test, SoftAutoCommitTest, SynonymTokenizerTest, TestBM25SimilarityFactory, TestConfig, DeleteShardTest, HdfsChaosMonkeySafeLeaderTest, HdfsDirectoryTest, UnloadDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, TestRandomDVFaceting, TestDistributedSearch]
   [junit4] Completed on J0 in 14.16s, 1 test, 1 failure <<< FAILURES!

[...truncated 521 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 362 suites, 1563 tests, 1 error, 1 failure, 35 ignored (7 assumptions)

Total time: 50 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b04 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure