You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/01/25 00:13:57 UTC

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b03) - Build # 9122 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9122/
Java: 64bit/jdk1.7.0_60-ea-b03 -XX:+UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 20296 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:64: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:162: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.

Total time: 47 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b03 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (64bit/ibm-j9-jdk6) - Build # 9125 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9125/
Java: 64bit/ibm-j9-jdk6 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

2 tests failed.
REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection

Error Message:
expected:<1> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<3>
	at __randomizedtesting.SeedInfo.seed([DC6FE7DC02A61F7E:875F1C1401E1BC02]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
	at java.lang.reflect.Method.invoke(Method.java:611)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:738)


REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([DC6FE7DC02A61F7E:BDFA3D3CEA771588]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded(SignatureUpdateProcessorFactoryTest.java:222)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
	at java.lang.reflect.Method.invoke(Method.java:611)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:738)




Build Log:
[...truncated 9576 lines...]
   [junit4] Suite: org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest
   [junit4]   2> 342127 T410 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979
   [junit4]   2> 342178 T410 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 342187 T410 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 342190 T410 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 342321 T410 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 342380 T410 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 342382 T410 oass.IndexSchema.readSchema Reading Solr Schema from schema12.xml
   [junit4]   2> 342389 T410 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 342777 T410 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 342780 T410 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 342788 T410 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 342829 T410 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 342876 T410 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 342880 T410 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 342880 T410 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 342881 T410 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 342902 T410 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 342903 T410 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 342903 T410 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 342947 T410 oasc.CoreContainer.<init> New CoreContainer 1808690126
   [junit4]   2> 342947 T410 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 342949 T410 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 342949 T410 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 342950 T410 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 342950 T410 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 342951 T410 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 342951 T410 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 342952 T410 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 342952 T410 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 342953 T410 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 342957 T410 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 342958 T410 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 342959 T410 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 342964 T411 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 342964 T411 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 342966 T411 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 342966 T411 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 343023 T411 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 343077 T411 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 343078 T411 oass.IndexSchema.readSchema Reading Solr Schema from schema12.xml
   [junit4]   2> 343084 T411 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 343436 T411 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 343439 T411 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 343441 T411 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 343464 T411 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 343468 T411 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 343472 T411 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 343473 T411 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979/
   [junit4]   2> 343474 T411 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@57295729
   [junit4]   2> 343475 T411 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979
   [junit4]   2> 343475 T411 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979/index/
   [junit4]   2> 343476 T411 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979/index' doesn't exist. Creating new index...
   [junit4]   2> 343477 T411 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979/index
   [junit4]   2> 343477 T411 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10976014061329531]
   [junit4]   2> 343478 T411 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_1,generation=1}
   [junit4]   2> 343479 T411 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 343479 T411 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 343486 T411 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 343486 T411 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 343487 T411 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 343487 T411 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 343488 T411 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 343488 T411 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 343489 T411 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 343489 T411 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 343490 T411 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 343491 T411 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 343491 T411 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 343492 T411 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 343492 T411 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 343493 T411 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 343493 T411 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 343494 T411 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 343495 T411 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 343495 T411 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 343496 T411 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 343497 T411 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 343498 T411 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 343498 T411 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 343499 T411 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 343500 T411 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 343500 T411 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 343501 T411 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 343502 T411 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 343503 T411 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 343503 T411 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 343504 T411 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 343505 T411 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 343506 T411 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 343506 T411 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 343507 T411 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 343508 T411 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 343509 T411 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 343510 T411 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 343512 T411 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 343513 T411 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 343514 T411 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 343515 T411 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.590639636415281]
   [junit4]   2> 343516 T411 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_1,generation=1}
   [junit4]   2> 343516 T411 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 343517 T411 oass.SolrIndexSearcher.<init> Opening Searcher@75ea75ea main
   [junit4]   2> 343518 T411 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 343525 T411 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 343597 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@75ea75ea main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 343604 T411 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 343606 T410 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 343609 T410 oas.SolrTestCaseJ4.setUp ###Starting testNonStringFieldsValues
   [junit4]   2> ASYNC  NEW_CORE C2829 name=collection1 org.apache.solr.core.SolrCore@f160f16
   [junit4]   2> 343610 T410 C2829 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 343611 T410 C2829 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 343615 T410 C2829 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343617 T410 C2829 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_2,generation=2}
   [junit4]   2> 343617 T410 C2829 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 343618 T410 C2829 oass.SolrIndexSearcher.<init> Opening Searcher@6e246e24 main
   [junit4]   2> 343619 T410 C2829 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 343622 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e246e24 main{StandardDirectoryReader(segments_2:2:nrt)}
   [junit4]   2> 343622 T410 C2829 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 7
   [junit4]   2> 343627 T410 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 343639 T410 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343641 T410 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_3,generation=3}
   [junit4]   2> 343642 T410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 343644 T410 oass.SolrIndexSearcher.<init> Opening Searcher@30ab30ab main
   [junit4]   2> 343645 T410 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 343647 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@30ab30ab main{StandardDirectoryReader(segments_3:4:nrt _0(4.7):c5/1:delGen=1)}
   [junit4]   2> 343648 T410 oas.SolrTestCaseJ4.tearDown ###Ending testNonStringFieldsValues
   [junit4]   2> 343657 T410 oas.SolrTestCaseJ4.setUp ###Starting testDupeAllFieldsDetection
   [junit4]   2> 343658 T410 C2829 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 343659 T410 C2829 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 343661 T410 C2829 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343662 T410 C2829 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_3,generation=3}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_4,generation=4}
   [junit4]   2> 343663 T410 C2829 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
   [junit4]   2> 343664 T410 C2829 oass.SolrIndexSearcher.<init> Opening Searcher@4ad04ad main
   [junit4]   2> 343665 T410 C2829 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 343675 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ad04ad main{StandardDirectoryReader(segments_4:5:nrt)}
   [junit4]   2> ASYNC  NEW_CORE C2830 name=collection1 org.apache.solr.core.SolrCore@f160f16
   [junit4]   2> 343676 T410 C2830 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 15
   [junit4]   2> 343682 T410 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343684 T410 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_4,generation=4}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_5,generation=5}
   [junit4]   2> 343685 T410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
   [junit4]   2> 343687 T410 oass.SolrIndexSearcher.<init> Opening Searcher@5900590 main
   [junit4]   2> 343687 T410 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 343690 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5900590 main{StandardDirectoryReader(segments_5:7:nrt _1(4.7):c3)}
   [junit4]   2> 343691 T410 oas.SolrTestCaseJ4.tearDown ###Ending testDupeAllFieldsDetection
   [junit4]   2> 343699 T410 oas.SolrTestCaseJ4.setUp ###Starting testFailNonIndexedSigWithOverwriteDupes
   [junit4]   2> 343700 T410 C2830 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 343701 T410 C2830 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 343702 T410 C2830 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343703 T410 C2830 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_5,generation=5}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_6,generation=6}
   [junit4]   2> 343704 T410 C2830 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
   [junit4]   2> 343705 T410 C2830 oass.SolrIndexSearcher.<init> Opening Searcher@493a493a main
   [junit4]   2> 343706 T410 C2830 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 343708 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@493a493a main{StandardDirectoryReader(segments_6:8:nrt)}
   [junit4]   2> 343709 T410 C2830 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 7
   [junit4]   2> 343711 T410 oas.SolrTestCaseJ4.tearDown ###Ending testFailNonIndexedSigWithOverwriteDupes
   [junit4]   2> 343719 T410 oas.SolrTestCaseJ4.setUp ###Starting testDupeDetection
   [junit4]   2> 343720 T410 C2830 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 343721 T410 C2830 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 343722 T410 C2830 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343723 T410 C2830 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_6,generation=6}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_7,generation=7}
   [junit4]   2> 343724 T410 C2830 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
   [junit4]   2> 343725 T410 C2830 oass.SolrIndexSearcher.<init> Opening Searcher@66476647 main
   [junit4]   2> 343726 T410 C2830 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 343729 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@66476647 main{StandardDirectoryReader(segments_7:9:nrt)}
   [junit4]   2> 343730 T410 C2830 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 8
   [junit4]   2> 343832 T410 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343882 T410 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_7,generation=7}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_8,generation=8}
   [junit4]   2> 343884 T410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 8
   [junit4]   2> 343885 T410 oass.SolrIndexSearcher.<init> Opening Searcher@73bc73bc main
   [junit4]   2> 343891 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73bc73bc main{StandardDirectoryReader(segments_8:11:nrt _2(4.7):c2)}
   [junit4]   2> 343892 T410 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 343897 T410 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343901 T410 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_8,generation=8}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_9,generation=9}
   [junit4]   2> 343902 T410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 9
   [junit4]   2> 343903 T410 oass.SolrIndexSearcher.<init> Opening Searcher@6b306b3 main
   [junit4]   2> 343908 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b306b3 main{StandardDirectoryReader(segments_9:13:nrt _2(4.7):c2 _3(4.7):c1)}
   [junit4]   2> 343909 T410 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 343911 T410 oas.SolrTestCaseJ4.tearDown ###Ending testDupeDetection
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SignatureUpdateProcessorFactoryTest -Dtests.method=testDupeDetection -Dtests.seed=DC6FE7DC02A61F7E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sl -Dtests.timezone=CTT -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.23s J1 | SignatureUpdateProcessorFactoryTest.testDupeDetection <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1> but was:<3>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DC6FE7DC02A61F7E:875F1C1401E1BC02]:0)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:738)
   [junit4]   2> 343955 T410 oas.SolrTestCaseJ4.setUp ###Starting testNonIndexedSignatureField
   [junit4]   2> ASYNC  NEW_CORE C2831 name=collection1 org.apache.solr.core.SolrCore@f160f16
   [junit4]   2> 343956 T410 C2831 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 343957 T410 C2831 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 2
   [junit4]   2> 343959 T410 C2831 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343960 T410 C2831 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_9,generation=9}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_a,generation=10}
   [junit4]   2> 343962 T410 C2831 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 10
   [junit4]   2> 343963 T410 C2831 oass.SolrIndexSearcher.<init> Opening Searcher@12461246 main
   [junit4]   2> 343964 T410 C2831 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 343967 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12461246 main{StandardDirectoryReader(segments_a:14:nrt)}
   [junit4]   2> 343968 T410 C2831 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 9
   [junit4]   2> 343974 T410 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 343976 T410 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_a,generation=10}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_b,generation=11}
   [junit4]   2> 343977 T410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 11
   [junit4]   2> 344029 T410 oass.SolrIndexSearcher.<init> Opening Searcher@2c182c18 main
   [junit4]   2> 344030 T410 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344032 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c182c18 main{StandardDirectoryReader(segments_b:16:nrt _4(4.7):c2)}
   [junit4]   2> 344033 T410 oas.SolrTestCaseJ4.tearDown ###Ending testNonIndexedSignatureField
   [junit4]   2> 344042 T410 oas.SolrTestCaseJ4.setUp ###Starting testMultiThreaded
   [junit4]   2> 344043 T410 C2831 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 344043 T410 C2831 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 344045 T410 C2831 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344046 T410 C2831 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_b,generation=11}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_c,generation=12}
   [junit4]   2> 344047 T410 C2831 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 12
   [junit4]   2> 344047 T410 C2831 oass.SolrIndexSearcher.<init> Opening Searcher@427f427f main
   [junit4]   2> 344048 T410 C2831 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344051 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@427f427f main{StandardDirectoryReader(segments_c:17:nrt)}
   [junit4]   2> 344052 T410 C2831 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 7
   [junit4]   2> 344066 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344103 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_c,generation=12}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_d,generation=13}
   [junit4]   2> 344104 T422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 13
   [junit4]   2> 344251 T422 oass.SolrIndexSearcher.<init> Opening Searcher@7e987e98 main
   [junit4]   2> 344256 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e987e98 main{StandardDirectoryReader(segments_d:35:nrt _8(4.7):c43/42:delGen=1 _a(4.7):c22/21:delGen=1 _9(4.7):c11/9:delGen=1)}
   [junit4]   2> 344257 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344258 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344501 T421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_d,generation=13}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_e,generation=14}
   [junit4]   2> 344502 T421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 14
   [junit4]   2> 344507 T421 oass.SolrIndexSearcher.<init> Opening Searcher@20942094 main
   [junit4]   2> 344508 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344509 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344511 T423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_e,generation=14}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_f,generation=15}
   [junit4]   2> 344512 T423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 15
   [junit4]   2> 344512 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@20942094 main{StandardDirectoryReader(segments_e:43:nrt _d(4.7):c58/56:delGen=3 _e(4.7):c2)}
   [junit4]   2> 344578 T423 oass.SolrIndexSearcher.<init> Opening Searcher@39443944 main
   [junit4]   2> 344580 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344581 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344582 T422 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 344583 T422 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 344584 T422 oass.SolrIndexSearcher.<init> Opening Searcher@20762076 main
   [junit4]   2> 344585 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344602 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344624 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39443944 main{StandardDirectoryReader(segments_e:43:nrt _d(4.7):c58/56:delGen=3 _e(4.7):c2)}
   [junit4]   2> 344644 T421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_f,generation=15}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_g,generation=16}
   [junit4]   2> 344648 T421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 16
   [junit4]   2> 344678 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@20762076 main{StandardDirectoryReader(segments_e:43:nrt _d(4.7):c58/56:delGen=3 _e(4.7):c2:delGen=1)}
   [junit4]   2> 344708 T421 oass.SolrIndexSearcher.<init> Opening Searcher@63356335 main
   [junit4]   2> 344710 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344711 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344712 T423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_g,generation=16}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_h,generation=17}
   [junit4]   2> 344713 T423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 17
   [junit4]   2> 344714 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63356335 main{StandardDirectoryReader(segments_g:50:nrt _d(4.7):c58/57:delGen=4 _e(4.7):c2/1:delGen=1 _g(4.7):c2)}
   [junit4]   2> 344715 T423 oass.SolrIndexSearcher.<init> Opening Searcher@12e812e8 main
   [junit4]   2> 344716 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344717 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344717 T422 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 344718 T422 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 344721 T422 oass.SolrIndexSearcher.<init> Opening Searcher@69a969a9 main
   [junit4]   2> 344724 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12e812e8 main{StandardDirectoryReader(segments_g:50:nrt _d(4.7):c58/57:delGen=4 _e(4.7):c2/1:delGen=1 _g(4.7):c2)}
   [junit4]   2> 344726 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344729 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344738 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69a969a9 main{StandardDirectoryReader(segments_h:53:nrt _d(4.7):c58/57:delGen=4 _e(4.7):c2/1:delGen=1 _g(4.7):c2/1 _h(4.7):c1)}
   [junit4]   2> 344774 T423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_h,generation=17}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_i,generation=18}
   [junit4]   2> 344775 T423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 18
   [junit4]   2> 344806 T423 oass.SolrIndexSearcher.<init> Opening Searcher@16431643 main
   [junit4]   2> 344818 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16431643 main{StandardDirectoryReader(segments_i:61:nrt _d(4.7):c58/57:delGen=4 _e(4.7):c2/1:delGen=1 _j(4.7):c1)}
   [junit4]   2> 344820 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344824 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344825 T421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_i,generation=18}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_j,generation=19}
   [junit4]   2> 344826 T421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 19
   [junit4]   2> 344827 T421 oass.SolrIndexSearcher.<init> Opening Searcher@144d144d main
   [junit4]   2> 344828 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344833 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@144d144d main{StandardDirectoryReader(segments_i:61:nrt _d(4.7):c58/57:delGen=4 _e(4.7):c2/1:delGen=1 _j(4.7):c1)}
   [junit4]   2> 344834 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344835 T422 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 344836 T422 oass.SolrIndexSearcher.<init> Opening Searcher@72217221 main
   [junit4]   2> 344837 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 344975 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 344979 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@72217221 main{StandardDirectoryReader(segments_i:61:nrt _d(4.7):c58/57:delGen=4 _e(4.7):c2/1:delGen=1 _j(4.7):c1)}
   [junit4]   2> 344989 T421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_j,generation=19}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_k,generation=20}
   [junit4]   2> 344990 T421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 20
   [junit4]   2> 345086 T421 oass.SolrIndexSearcher.<init> Opening Searcher@3be93be9 main
   [junit4]   2> 345088 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345091 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3be93be9 main{StandardDirectoryReader(segments_k:68:nrt _k(4.7):c2/1 _l(4.7):c1)}
   [junit4]   2> 345093 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345094 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_k,generation=20}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_l,generation=21}
   [junit4]   2> 345095 T422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 21
   [junit4]   2> 345096 T422 oass.SolrIndexSearcher.<init> Opening Searcher@687d687d main
   [junit4]   2> 345097 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345098 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345099 T423 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 345101 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@687d687d main{StandardDirectoryReader(segments_k:68:nrt _k(4.7):c2/1:delGen=1 _l(4.7):c1)}
   [junit4]   2> 345102 T423 oass.SolrIndexSearcher.<init> Opening Searcher@bcf0bcf main
   [junit4]   2> 345103 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345106 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bcf0bcf main{StandardDirectoryReader(segments_k:68:nrt _k(4.7):c2/1:delGen=1 _l(4.7):c1)}
   [junit4]   2> 345107 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345120 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_l,generation=21}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_m,generation=22}
   [junit4]   2> 345121 T422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 22
   [junit4]   2> 345127 T422 oass.SolrIndexSearcher.<init> Opening Searcher@7da97da9 main
   [junit4]   2> 345132 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7da97da9 main{StandardDirectoryReader(segments_m:75:nrt _m(4.7):c2/1 _n(4.7):c1)}
   [junit4]   2> 345134 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345137 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345142 T421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_m,generation=22}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_n,generation=23}
   [junit4]   2> 345143 T421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 23
   [junit4]   2> 345145 T421 oass.SolrIndexSearcher.<init> Opening Searcher@125b125b main
   [junit4]   2> 345146 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345147 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345148 T423 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 345150 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@125b125b main{StandardDirectoryReader(segments_n:79:nrt _o(4.7):c1)}
   [junit4]   2> 345151 T423 oass.SolrIndexSearcher.<init> Opening Searcher@5da85da8 main
   [junit4]   2> 345155 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345203 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345207 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_n,generation=23}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_o,generation=24}
   [junit4]   2> 345208 T422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 24
   [junit4]   2> 345361 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5da85da8 main{StandardDirectoryReader(segments_n:79:nrt _o(4.7):c1)}
   [junit4]   2> 345362 T422 oass.SolrIndexSearcher.<init> Opening Searcher@57a457a4 main
   [junit4]   2> 345363 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345363 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345364 T421 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 345365 T421 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 345366 T421 oass.SolrIndexSearcher.<init> Opening Searcher@29f829f8 main
   [junit4]   2> 345367 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345368 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345372 T423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_o,generation=24}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_p,generation=25}
   [junit4]   2> 345373 T423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 25
   [junit4]   2> 345375 T423 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=3
   [junit4]   2> 345376 T423 oass.SolrIndexSearcher.<init> Opening Searcher@7b247b24 main
   [junit4]   2> 345377 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345382 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@57a457a4 main{StandardDirectoryReader(segments_o:81:nrt _o(4.7):c1 _p(4.7):c1)}
   [junit4]   2> 345385 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345413 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_p,generation=25}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_q,generation=26}
   [junit4]   2> 345414 T422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 26
   [junit4]   2> 345454 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29f829f8 main{StandardDirectoryReader(segments_o:81:nrt _o(4.7):c1 _p(4.7):c1)}
   [junit4]   2> 345454 T422 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 345456 T422 oass.SolrIndexSearcher.<init> Opening Searcher@21cc21cc main
   [junit4]   2> 345457 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345469 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345474 T421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_q,generation=26}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_r,generation=27}
   [junit4]   2> 345475 T421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 27
   [junit4]   2> 345476 T421 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=3
   [junit4]   2> 345476 T421 oass.SolrIndexSearcher.<init> Opening Searcher@74f074f main
   [junit4]   2> 345477 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345482 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b247b24 main{StandardDirectoryReader(segments_p:85:nrt _o(4.7):c1 _q(4.7):c1)}
   [junit4]   2> 345492 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345498 T423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_r,generation=27}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_s,generation=28}
   [junit4]   2> 345499 T423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 28
   [junit4]   2> 345501 T423 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=3
   [junit4]   2> 345502 T423 oass.SolrIndexSearcher.<init> Opening Searcher@13d113d1 main
   [junit4]   2> 345507 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345518 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21cc21cc main{StandardDirectoryReader(segments_q:89:nrt _q(4.7):c1 _r(4.7):c1)}
   [junit4]   2> 345530 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345534 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_s,generation=28}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_t,generation=29}
   [junit4]   2> 345535 T422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 29
   [junit4]   2> 345536 T422 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=3
   [junit4]   2> 345536 T422 oass.SolrIndexSearcher.<init> Opening Searcher@21262126 main
   [junit4]   2> 345537 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345541 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@74f074f main{StandardDirectoryReader(segments_r:93:nrt _q(4.7):c1 _s(4.7):c1)}
   [junit4]   2> 345554 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345560 T421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_t,generation=29}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_u,generation=30}
   [junit4]   2> 345623 T421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 30
   [junit4]   2> 345624 T412 oasc.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo WARN Could not getStatistics on info bean org.apache.solr.search.SolrIndexSearcher org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
   [junit4]   2> 	at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
   [junit4]   2> 	at org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
   [junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2251)
   [junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo(JmxMonitoredMap.java:236)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1792)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1634)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1801)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:518)
   [junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:138)
   [junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
   [junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:310)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1864)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.access$100(SolrCore.java:133)
   [junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1738)
   [junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:738)
   [junit4]   2> 
   [junit4]   2> 345637 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13d113d1 main{StandardDirectoryReader(segments_s:97:nrt _t(4.7):c1)}
   [junit4]   2> 345640 T421 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 345661 T421 oass.SolrIndexSearcher.<init> Opening Searcher@bc60bc6 main
   [junit4]   2> 345662 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345667 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345673 T423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_u,generation=30}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_v,generation=31}
   [junit4]   2> 345674 T423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 31
   [junit4]   2> 345685 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21262126 main{StandardDirectoryReader(segments_t:101:nrt _u(4.7):c1)}
   [junit4]   2> 345687 T423 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 345688 T423 oass.SolrIndexSearcher.<init> Opening Searcher@47554755 main
   [junit4]   2> 345689 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345695 T422 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345699 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_v,generation=31}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_w,generation=32}
   [junit4]   2> 345700 T422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 32
   [junit4]   2> 345703 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bc60bc6 main{StandardDirectoryReader(segments_u:103:nrt _u(4.7):c1 _v(4.7):c1)}
   [junit4]   2> 345704 T422 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 345706 T422 oass.SolrIndexSearcher.<init> Opening Searcher@77ac77ac main
   [junit4]   2> 345707 T412 oasc.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo WARN Could not getStatistics on info bean org.apache.solr.search.SolrIndexSearcher org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
   [junit4]   2> 	at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253)
   [junit4]   2> 	at org.apache.lucene.index.StandardDirectoryReader.getVersion(StandardDirectoryReader.java:331)
   [junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.getStatistics(SolrIndexSearcher.java:2251)
   [junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo(JmxMonitoredMap.java:236)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1792)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1634)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1801)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:518)
   [junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:138)
   [junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
   [junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:310)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1864)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.access$100(SolrCore.java:133)
   [junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1738)
   [junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:738)
   [junit4]   2> 
   [junit4]   2> 345712 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47554755 main{StandardDirectoryReader(segments_v:107:nrt _u(4.7):c1 _w(4.7):c1)}
   [junit4]   2> 345713 T422 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345714 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345717 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@77ac77ac main{StandardDirectoryReader(segments_w:109:nrt _u(4.7):c1 _w(4.7):c1 _x(4.7):c1)}
   [junit4]   2> 345777 T421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_w,generation=32}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_x,generation=33}
   [junit4]   2> 345779 T421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 33
   [junit4]   2> 345780 T421 oass.SolrIndexSearcher.<init> Opening Searcher@263d263d main
   [junit4]   2> 345781 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345782 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345783 T423 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 345784 T423 oasc.SolrCore.getSearcher WARN [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 345785 T423 oass.SolrIndexSearcher.<init> Opening Searcher@25232523 main
   [junit4]   2> 345786 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345794 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@263d263d main{StandardDirectoryReader(segments_x:113:nrt _w(4.7):c1 _x(4.7):c1 _y(4.7):c2/1:delGen=1)}
   [junit4]   2> 345798 T421 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345798 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@25232523 main{StandardDirectoryReader(segments_x:113:nrt _w(4.7):c1 _x(4.7):c1 _y(4.7):c2/1:delGen=1)}
   [junit4]   2> 345856 T421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_x,generation=33}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_y,generation=34}
   [junit4]   2> 345857 T421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 34
   [junit4]   2> 345861 T421 oass.SolrIndexSearcher.<init> Opening Searcher@61b761b7 main
   [junit4]   2> 345869 T421 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345870 T423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345872 T423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_y,generation=34}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@36573657 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40064006)),segFN=segments_z,generation=35}
   [junit4]   2> 345873 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@61b761b7 main{StandardDirectoryReader(segments_y:121:nrt _w(4.7):c1 _10(4.7):c1)}
   [junit4]   2> 345873 T423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 35
   [junit4]   2> 345925 T423 oass.SolrIndexSearcher.<init> Opening Searcher@768a768a main
   [junit4]   2> 345929 T423 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345932 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@768a768a main{StandardDirectoryReader(segments_y:121:nrt _w(4.7):c1 _10(4.7):c1)}
   [junit4]   2> ASYNC  NEW_CORE C2832 name=collection1 org.apache.solr.core.SolrCore@f160f16
   [junit4]   2> 345934 T410 C2832 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 345935 T410 C2832 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 345936 T410 C2832 oass.SolrIndexSearcher.<init> Opening Searcher@19821982 main
   [junit4]   2> 345937 T410 C2832 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 345941 T412 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19821982 main{StandardDirectoryReader(segments_y:121:nrt _w(4.7):c1 _10(4.7):c1)}
   [junit4]   2> 345942 T410 C2832 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 8
   [junit4]   2> 345943 T410 oas.SolrTestCaseJ4.tearDown ###Ending testMultiThreaded
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SignatureUpdateProcessorFactoryTest -Dtests.method=testMultiThreaded -Dtests.seed=DC6FE7DC02A61F7E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sl -Dtests.timezone=CTT -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.92s J1 | SignatureUpdateProcessorFactoryTest.testMultiThreaded <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<1> but was:<2>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DC6FE7DC02A61F7E:BDFA3D3CEA771588]:0)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
   [junit4]    > 	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded(SignatureUpdateProcessorFactoryTest.java:222)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:738)
   [junit4]   2> 345957 T410 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 345957 T410 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1808690126
   [junit4]   2> 345958 T410 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@f160f16
   [junit4]   2> 346129 T410 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=42,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=253,cumulative_deletesById=0,cumulative_deletesByQuery=6,cumulative_errors=0}
   [junit4]   2> 346205 T410 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 346218 T410 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 346219 T410 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 346232 T410 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 346233 T410 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 346233 T410 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979/index;done=false>>]
   [junit4]   2> 346234 T410 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979/index
   [junit4]   2> 346234 T410 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979;done=false>>]
   [junit4]   2> 346235 T410 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SignatureUpdateProcessorFactoryTest-1390623222979
   [junit4]   2> NOTE: test params are: codec=Appending, sim=DefaultSimilarity, locale=sl, timezone=CTT
   [junit4]   2> NOTE: Linux 3.8.0-35-generic amd64/IBM Corporation 1.6.0 (64-bit)/cpus=8,threads=1,free=11274960,total=43829248
   [junit4]   2> NOTE: All tests run in this JVM: [SolrPluginUtilsTest, TestCollationFieldDocValues, JSONWriterTest, CursorPagingTest, TestUpdate, DirectSolrSpellCheckerTest, SimplePostToolTest, TestReload, TestOmitPositions, TestStressRecovery, HardAutoCommitTest, SuggesterWFSTTest, SpellingQueryConverterTest, OpenExchangeRatesOrgProviderTest, TestQuerySenderNoQuery, TestRTGBase, DirectSolrConnectionTest, LeaderElectionIntegrationTest, TestAtomicUpdateErrorCases, NotRequiredUniqueKeyTest, TestArbitraryIndexDir, TestWordDelimiterFilterFactory, TestFastWriter, HdfsBasicDistributedZkTest, CollectionsAPIDistributedZkTest, TestHighFrequencyDictionaryFactory, DocumentAnalysisRequestHandlerTest, TestUniqueKeyFieldResource, AddBlockUpdateTest, ZkNodePropsTest, SignatureUpdateProcessorFactoryTest]
   [junit4] Completed on J1 in 4.24s, 6 tests, 2 failures <<< FAILURES!

[...truncated 714 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:452: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 355 suites, 1546 tests, 2 failures, 67 ignored (7 assumptions)

Total time: 39 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/ibm-j9-jdk6 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build # 9124 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9124/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:40401/ga_jc/collection2, http://127.0.0.1:46238/ga_jc/collection2, http://127.0.0.1:43829/ga_jc/collection2, http://127.0.0.1:46209/ga_jc/collection2, http://127.0.0.1:39088/ga_jc/collection2]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:40401/ga_jc/collection2, http://127.0.0.1:46238/ga_jc/collection2, http://127.0.0.1:43829/ga_jc/collection2, http://127.0.0.1:46209/ga_jc/collection2, http://127.0.0.1:39088/ga_jc/collection2]
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:352)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:410)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:143)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: no servers hosting shard: 
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:317)
	... 46 more




Build Log:
[...truncated 10491 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> 759856 T2775 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ga_jc/
   [junit4]   2> 759859 T2775 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1390617520913
   [junit4]   2> 759859 T2775 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 759860 T2776 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 759960 T2775 oasc.ZkTestServer.run start zk server on port:60773
   [junit4]   2> 759961 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 759964 T2782 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e7585c name:ZooKeeperConnection Watcher:127.0.0.1:60773 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 759965 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 759965 T2775 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 759968 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 759969 T2784 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3acce4 name:ZooKeeperConnection Watcher:127.0.0.1:60773/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 759970 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 759970 T2775 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 759973 T2775 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 759975 T2775 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 759977 T2775 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 759980 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 759980 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 759984 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 759985 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 759988 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 759988 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 759990 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 759991 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 759993 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 759993 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 759995 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 759996 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 759997 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 759998 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 760001 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 760001 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 760004 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 760004 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 760007 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 760007 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 760010 T2775 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 760010 T2775 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 760076 T2775 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 760080 T2775 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40401
   [junit4]   2> 760080 T2775 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 760081 T2775 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 760081 T2775 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390617521067
   [junit4]   2> 760081 T2775 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390617521067/'
   [junit4]   2> 760091 T2775 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390617521067/solr.xml
   [junit4]   2> 760131 T2775 oasc.CoreContainer.<init> New CoreContainer 16065525
   [junit4]   2> 760131 T2775 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390617521067/]
   [junit4]   2> 760132 T2775 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 760133 T2775 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 760133 T2775 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 760133 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 760133 T2775 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 760134 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 760134 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 760134 T2775 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 760134 T2775 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 760136 T2775 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 760137 T2775 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 760137 T2775 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 760137 T2775 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60773/solr
   [junit4]   2> 760138 T2775 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 760138 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 760140 T2795 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bc2ef name:ZooKeeperConnection Watcher:127.0.0.1:60773 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 760140 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 760142 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 760143 T2797 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bb4dbb name:ZooKeeperConnection Watcher:127.0.0.1:60773/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 760144 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 760145 T2775 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 760148 T2775 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 760150 T2775 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 760151 T2775 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40401_ga_jc
   [junit4]   2> 760152 T2775 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40401_ga_jc
   [junit4]   2> 760155 T2775 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 760157 T2775 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 760162 T2775 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 760164 T2775 oasc.Overseer.start Overseer (id=91135509850685443-127.0.0.1:40401_ga_jc-n_0000000000) starting
   [junit4]   2> 760167 T2775 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 760170 T2799 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 760170 T2775 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 760171 T2799 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 760173 T2775 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 760174 T2775 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 760177 T2798 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 760180 T2800 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 760180 T2800 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 760182 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760182 T2800 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 760183 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 760183 T2798 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40401/ga_jc",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40401_ga_jc",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 760184 T2798 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 760184 T2798 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 760186 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 760186 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 761182 T2800 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 761182 T2800 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390617521067/collection1
   [junit4]   2> 761183 T2800 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 761183 T2800 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 761184 T2800 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 761185 T2800 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390617521067/collection1/'
   [junit4]   2> 761185 T2800 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390617521067/collection1/lib/classes/' to classloader
   [junit4]   2> 761185 T2800 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390617521067/collection1/lib/README' to classloader
   [junit4]   2> 761218 T2800 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 761260 T2800 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 761261 T2800 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 761266 T2800 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 761575 T2800 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 761577 T2800 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 761578 T2800 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 761581 T2800 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 761599 T2800 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 761600 T2800 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390617521067/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/control/data/
   [junit4]   2> 761600 T2800 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@eccfb2
   [junit4]   2> 761600 T2800 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/control/data
   [junit4]   2> 761601 T2800 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/control/data/index/
   [junit4]   2> 761601 T2800 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 761601 T2800 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/control/data/index
   [junit4]   2> 761601 T2800 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=84.466796875, floorSegmentMB=0.3203125, forceMergeDeletesPctAllowed=18.79491754232937, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5392669939282948
   [junit4]   2> 761602 T2800 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@b5f7c0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b1294b),segFN=segments_1,generation=1}
   [junit4]   2> 761602 T2800 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 761604 T2800 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 761605 T2800 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 761605 T2800 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 761605 T2800 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 761605 T2800 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 761605 T2800 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 761606 T2800 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 761606 T2800 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 761606 T2800 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 761607 T2800 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 761607 T2800 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 761607 T2800 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 761607 T2800 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 761608 T2800 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 761608 T2800 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 761608 T2800 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 761610 T2800 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 761613 T2800 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 761613 T2800 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 761613 T2800 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=14.845703125, floorSegmentMB=1.0322265625, forceMergeDeletesPctAllowed=27.176144040908753, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 761614 T2800 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@b5f7c0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b1294b),segFN=segments_1,generation=1}
   [junit4]   2> 761614 T2800 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 761614 T2800 oass.SolrIndexSearcher.<init> Opening Searcher@1e8f84f main
   [junit4]   2> 761616 T2801 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e8f84f main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 761617 T2800 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 761618 T2775 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 761618 T2775 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 761618 T2804 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40401/ga_jc collection:control_collection shard:shard1
   [junit4]   2> 761619 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 761620 T2804 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 761620 T2806 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@547457 name:ZooKeeperConnection Watcher:127.0.0.1:60773/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 761621 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 761622 T2775 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 761624 T2775 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 761628 T2804 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 761631 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761631 T2804 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 761632 T2804 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2365 name=collection1 org.apache.solr.core.SolrCore@478ea4 url=http://127.0.0.1:40401/ga_jc/collection1 node=127.0.0.1:40401_ga_jc C2365_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:40401/ga_jc, core=collection1, node_name=127.0.0.1:40401_ga_jc}
   [junit4]   2> 761632 T2804 C2365 P40401 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40401/ga_jc/collection1/
   [junit4]   2> 761632 T2804 C2365 P40401 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 761632 T2804 C2365 P40401 oasc.SyncStrategy.syncToMe http://127.0.0.1:40401/ga_jc/collection1/ has no replicas
   [junit4]   2> 761633 T2804 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40401/ga_jc/collection1/ shard1
   [junit4]   2> 761633 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 761633 T2804 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 761636 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761638 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761641 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761689 T2775 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 761689 T2775 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 761692 T2775 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43829
   [junit4]   2> 761692 T2775 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 761693 T2775 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 761693 T2775 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390617522679
   [junit4]   2> 761693 T2775 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390617522679/'
   [junit4]   2> 761705 T2775 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390617522679/solr.xml
   [junit4]   2> 761745 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 761746 T2806 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 761749 T2775 oasc.CoreContainer.<init> New CoreContainer 27171074
   [junit4]   2> 761749 T2775 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390617522679/]
   [junit4]   2> 761751 T2775 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 761751 T2775 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 761751 T2775 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 761752 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 761752 T2775 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 761752 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 761753 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 761753 T2775 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 761753 T2775 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 761755 T2775 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 761755 T2775 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 761756 T2775 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 761756 T2775 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60773/solr
   [junit4]   2> 761756 T2775 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 761757 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 761759 T2817 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bf0d09 name:ZooKeeperConnection Watcher:127.0.0.1:60773 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 761760 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 761762 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 761763 T2819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bbd28 name:ZooKeeperConnection Watcher:127.0.0.1:60773/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 761763 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 761766 T2775 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 761792 T2804 oasc.ZkController.register We are http://127.0.0.1:40401/ga_jc/collection1/ and leader is http://127.0.0.1:40401/ga_jc/collection1/
   [junit4]   2> 761793 T2804 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40401/ga_jc
   [junit4]   2> 761793 T2804 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 761793 T2804 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 761793 T2804 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 761795 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761795 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761795 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761795 T2804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 761796 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 761797 T2798 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40401/ga_jc",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40401_ga_jc",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 761799 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761900 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 761901 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 761901 T2806 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 762769 T2775 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43829_ga_jc
   [junit4]   2> 762770 T2775 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43829_ga_jc
   [junit4]   2> 762772 T2797 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 762773 T2806 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 762772 T2819 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 762778 T2820 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 762778 T2820 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 762779 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762780 T2820 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 762780 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762780 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762781 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 762782 T2798 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43829/ga_jc",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43829_ga_jc",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 762782 T2798 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 762782 T2798 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 762785 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762888 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 762888 T2806 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 762888 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 763780 T2820 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 763780 T2820 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390617522679/collection1
   [junit4]   2> 763781 T2820 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 763781 T2820 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 763782 T2820 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 763783 T2820 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390617522679/collection1/'
   [junit4]   2> 763784 T2820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390617522679/collection1/lib/classes/' to classloader
   [junit4]   2> 763784 T2820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390617522679/collection1/lib/README' to classloader
   [junit4]   2> 763813 T2820 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 763855 T2820 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 763857 T2820 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 763862 T2820 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 764170 T2820 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 764172 T2820 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 764173 T2820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 764176 T2820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 764196 T2820 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 764197 T2820 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390617522679/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty1/
   [junit4]   2> 764197 T2820 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@eccfb2
   [junit4]   2> 764197 T2820 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty1
   [junit4]   2> 764198 T2820 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty1/index/
   [junit4]   2> 764198 T2820 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 764198 T2820 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty1/index
   [junit4]   2> 764198 T2820 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=84.466796875, floorSegmentMB=0.3203125, forceMergeDeletesPctAllowed=18.79491754232937, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5392669939282948
   [junit4]   2> 764199 T2820 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@d61dd3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@18f5483),segFN=segments_1,generation=1}
   [junit4]   2> 764199 T2820 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 764202 T2820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 764202 T2820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 764203 T2820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 764203 T2820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 764203 T2820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 764203 T2820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 764204 T2820 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 764204 T2820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 764204 T2820 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 764204 T2820 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 764205 T2820 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 764205 T2820 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 764205 T2820 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 764206 T2820 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 764206 T2820 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 764207 T2820 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 764209 T2820 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 764211 T2820 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 764211 T2820 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 764211 T2820 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=14.845703125, floorSegmentMB=1.0322265625, forceMergeDeletesPctAllowed=27.176144040908753, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 764212 T2820 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@d61dd3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@18f5483),segFN=segments_1,generation=1}
   [junit4]   2> 764212 T2820 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 764212 T2820 oass.SolrIndexSearcher.<init> Opening Searcher@b2da4c main
   [junit4]   2> 764215 T2821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b2da4c main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 764217 T2820 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 764217 T2775 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 764218 T2775 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 764217 T2824 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43829/ga_jc collection:collection1 shard:shard3
   [junit4]   2> 764219 T2824 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 764224 T2824 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 764225 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764225 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764226 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764226 T2824 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 764226 T2824 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2366 name=collection1 org.apache.solr.core.SolrCore@9f620d url=http://127.0.0.1:43829/ga_jc/collection1 node=127.0.0.1:43829_ga_jc C2366_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:43829/ga_jc, core=collection1, node_name=127.0.0.1:43829_ga_jc}
   [junit4]   2> 764227 T2824 C2366 P43829 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43829/ga_jc/collection1/
   [junit4]   2> 764227 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 764227 T2824 C2366 P43829 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 764227 T2824 C2366 P43829 oasc.SyncStrategy.syncToMe http://127.0.0.1:43829/ga_jc/collection1/ has no replicas
   [junit4]   2> 764228 T2824 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43829/ga_jc/collection1/ shard3
   [junit4]   2> 764228 T2824 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 764230 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764232 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764235 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764292 T2775 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 764293 T2775 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 764295 T2775 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46209
   [junit4]   2> 764295 T2775 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 764296 T2775 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 764296 T2775 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390617525272
   [junit4]   2> 764296 T2775 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390617525272/'
   [junit4]   2> 764306 T2775 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390617525272/solr.xml
   [junit4]   2> 764338 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 764338 T2806 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 764338 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 764342 T2775 oasc.CoreContainer.<init> New CoreContainer 10048250
   [junit4]   2> 764342 T2775 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390617525272/]
   [junit4]   2> 764343 T2775 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 764344 T2775 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 764344 T2775 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 764344 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 764344 T2775 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 764345 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 764345 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 764345 T2775 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 764346 T2775 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 764348 T2775 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 764348 T2775 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 764348 T2775 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 764348 T2775 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60773/solr
   [junit4]   2> 764349 T2775 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 764349 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 764351 T2835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16d9f7 name:ZooKeeperConnection Watcher:127.0.0.1:60773 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 764351 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 764363 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 764365 T2837 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ce0ab5 name:ZooKeeperConnection Watcher:127.0.0.1:60773/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 764365 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 764368 T2775 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 764383 T2824 oasc.ZkController.register We are http://127.0.0.1:43829/ga_jc/collection1/ and leader is http://127.0.0.1:43829/ga_jc/collection1/
   [junit4]   2> 764383 T2824 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43829/ga_jc
   [junit4]   2> 764383 T2824 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 764383 T2824 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 764384 T2824 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 764385 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764385 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764385 T2824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 764385 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764387 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 764387 T2798 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43829/ga_jc",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43829_ga_jc",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 764389 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764491 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 764491 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 764491 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 764491 T2806 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 765370 T2775 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46209_ga_jc
   [junit4]   2> 765372 T2775 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46209_ga_jc
   [junit4]   2> 765375 T2819 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 765375 T2837 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 765375 T2797 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 765375 T2806 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 765381 T2838 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 765381 T2838 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 765383 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 765383 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 765383 T2838 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 765383 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 765384 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 765385 T2798 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46209/ga_jc",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46209_ga_jc",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 765385 T2798 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 765385 T2798 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 765388 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 765490 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 765490 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 765490 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 765490 T2806 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 766383 T2838 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 766384 T2838 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390617525272/collection1
   [junit4]   2> 766384 T2838 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 766384 T2838 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 766385 T2838 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 766386 T2838 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390617525272/collection1/'
   [junit4]   2> 766386 T2838 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390617525272/collection1/lib/classes/' to classloader
   [junit4]   2> 766387 T2838 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390617525272/collection1/lib/README' to classloader
   [junit4]   2> 766424 T2838 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 766455 T2838 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 766457 T2838 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 766462 T2838 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 766769 T2838 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 766771 T2838 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 766772 T2838 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 766775 T2838 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 766792 T2838 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 766792 T2838 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390617525272/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty2/
   [junit4]   2> 766792 T2838 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@eccfb2
   [junit4]   2> 766793 T2838 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty2
   [junit4]   2> 766793 T2838 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty2/index/
   [junit4]   2> 766793 T2838 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 766794 T2838 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty2/index
   [junit4]   2> 766794 T2838 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=84.466796875, floorSegmentMB=0.3203125, forceMergeDeletesPctAllowed=18.79491754232937, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5392669939282948
   [junit4]   2> 766794 T2838 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7632d7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f2357d),segFN=segments_1,generation=1}
   [junit4]   2> 766795 T2838 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 766797 T2838 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 766798 T2838 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 766798 T2838 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 766798 T2838 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 766798 T2838 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 766799 T2838 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 766799 T2838 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 766799 T2838 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 766799 T2838 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 766800 T2838 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 766800 T2838 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 766800 T2838 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 766801 T2838 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 766801 T2838 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 766801 T2838 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 766802 T2838 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 766804 T2838 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 766806 T2838 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 766807 T2838 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 766808 T2838 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=14.845703125, floorSegmentMB=1.0322265625, forceMergeDeletesPctAllowed=27.176144040908753, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 766808 T2838 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7632d7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f2357d),segFN=segments_1,generation=1}
   [junit4]   2> 766808 T2838 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 766809 T2838 oass.SolrIndexSearcher.<init> Opening Searcher@16acd96 main
   [junit4]   2> 766812 T2839 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16acd96 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 766814 T2838 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 766814 T2775 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 766815 T2775 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 766814 T2842 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46209/ga_jc collection:collection1 shard:shard2
   [junit4]   2> 766816 T2842 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 766821 T2842 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 766822 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766822 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766823 T2842 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 766823 T2842 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 766823 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C2367 name=collection1 org.apache.solr.core.SolrCore@10cb8af url=http://127.0.0.1:46209/ga_jc/collection1 node=127.0.0.1:46209_ga_jc C2367_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:46209/ga_jc, core=collection1, node_name=127.0.0.1:46209_ga_jc}
   [junit4]   2> 766823 T2842 C2367 P46209 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46209/ga_jc/collection1/
   [junit4]   2> 766824 T2842 C2367 P46209 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 766824 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 766824 T2842 C2367 P46209 oasc.SyncStrategy.syncToMe http://127.0.0.1:46209/ga_jc/collection1/ has no replicas
   [junit4]   2> 766824 T2842 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46209/ga_jc/collection1/ shard2
   [junit4]   2> 766824 T2842 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 766827 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766829 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766832 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766882 T2775 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 766883 T2775 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 766885 T2775 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35159
   [junit4]   2> 766885 T2775 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 766886 T2775 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 766886 T2775 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390617527869
   [junit4]   2> 766886 T2775 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390617527869/'
   [junit4]   2> 766896 T2775 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390617527869/solr.xml
   [junit4]   2> 766935 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 766935 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 766935 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 766935 T2806 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 766938 T2775 oasc.CoreContainer.<init> New CoreContainer 7660205
   [junit4]   2> 766939 T2775 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390617527869/]
   [junit4]   2> 766940 T2775 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 766940 T2775 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 766941 T2775 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 766941 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 766941 T2775 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 766941 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 766942 T2775 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 766942 T2775 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 766942 T2775 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 766944 T2775 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 766945 T2775 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 766945 T2775 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 766945 T2775 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60773/solr
   [junit4]   2> 766945 T2775 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 766946 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 766952 T2853 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@97dbaa name:ZooKeeperConnection Watcher:127.0.0.1:60773 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 766953 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 766954 T2775 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 766956 T2855 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1862b7a name:ZooKeeperConnection Watcher:127.0.0.1:60773/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 766956 T2775 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 766959 T2775 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 766980 T2842 oasc.ZkController.register We are http://127.0.0.1:46209/ga_jc/collection1/ and leader is http://127.0.0.1:46209/ga_jc/collection1/
   [junit4]   2> 766980 T2842 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46209/ga_jc
   [junit4]   2> 766980 T2842 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 766980 T2842 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 766981 T2842 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 766982 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766982 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766982 T2842 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 766982 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 766984 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 766984 T2798 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46209/ga_jc",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46209_ga_jc",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 766987 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 767089 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 767089 T2806 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 767089 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 767089 T2855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 767089 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 767962 T2775 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35159_ga_jc
   [junit4]   2> 767963 T2775 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35159_ga_jc
   [junit4]   2> 767966 T2837 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 767966 T2819 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 767966 T2855 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 767966 T2797 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 767966 T2806 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 767972 T2856 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 767972 T2856 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 767973 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 767973 T2856 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 767973 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 767973 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 767975 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 767975 T2798 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35159/ga_jc",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35159_ga_jc",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 767976 T2798 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 767976 T2798 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 767978 T2797 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 768080 T2806 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 768080 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 768080 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 768080 T2855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 768080 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 768973 T2856 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 768974 T2856 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390617527869/collection1
   [junit4]   2> 768974 T2856 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 768974 T2856 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 768975 T2856 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 768975 T2856 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390617527869/collection1/'
   [junit4]   2> 768976 T2856 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390617527869/collection1/lib/classes/' to classloader
   [junit4]   2> 768976 T2856 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390617527869/collection1/lib/README' to classloader
   [junit4]   2> 769013 T2856 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 769055 T2856 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 769057 T2856 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 769062 T2856 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 769355 T2856 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 769364 T2856 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 769366 T2856 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 769369 T2856 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 769387 T2856 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 769388 T2856 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390617527869/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty3/
   [junit4]   2> 769388 T2856 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@eccfb2
   [junit4]   2> 769388 T2856 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty3
   [junit4]   2> 769389 T2856 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty3/index/
   [junit4]   2> 769389 T2856 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 769389 T2856 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390617520913/jetty3/index
   [junit4]   2> 769390 T2856 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=84.466796875, floorSegmentMB=0.3203125, forceMergeDeletesPctAllowed=18.79491754232937, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5392669939282948
   [junit4]   2> 769390 T2856 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1b8b189 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c90df9),segFN=segments_1,generation=1}
   [junit4]   2> 769390 T2856 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 769393 T2856 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 769394 T2856 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 769394 T2856 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 769394 T2856 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 769394 T2856 oasup.UpdateRequestProce

[...truncated too long message...]

(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 844900 T2873 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 844901 T2873 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection2/leader_elect/shard33/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 844901 T2873 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection2/leader_elect/shard8/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 844902 T2873 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection2/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ko_KR, timezone=Europe/Busingen
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=161273592,total=323268608
   [junit4]   2> NOTE: All tests run in this JVM: [TestHashPartitioner, HdfsRecoveryZkTest, DistanceFunctionTest, CurrencyFieldXmlFileTest, TestFastWriter, CoreMergeIndexesAdminHandlerTest, TestQuerySenderListener, NoCacheHeaderTest, TestCodecSupport, TestCSVResponseWriter, HdfsUnloadDistributedZkTest, TestRecovery, CoreContainerCoreInitFailuresTest, TestIndexSearcher, TestSolrDeletionPolicy2, RequestHandlersTest, AddBlockUpdateTest, SortByFunctionTest, MultiTermTest, TestDistributedSearch, TestSweetSpotSimilarityFactory, SpellCheckCollatorTest, PluginInfoTest, TestSerializedLuceneMatchVersion, HdfsCollectionsAPIDistributedZkTest, OpenExchangeRatesOrgProviderTest, TestCollationKeyRangeQueries, HardAutoCommitTest, TestRangeQuery, TestManagedSchemaFieldResource, DocumentAnalysisRequestHandlerTest, TestStressLucene, TestFileDictionaryLookup, TimeZoneUtilsTest, TestCloudManagedSchemaAddField, TestCollationField, SolrTestCaseJ4Test, TestZkChroot, RAMDirectoryFactoryTest, TermVectorComponentDistributedTest, SpellCheckComponentTest, ZkNodePropsTest, TestReload, TestDynamicFieldResource, CurrencyFieldOpenExchangeTest, PathHierarchyTokenizerFactoryTest, ShardSplitTest, TestXIncludeConfig, StressHdfsTest, TestFreeTextSuggestions, SpellingQueryConverterTest, TestNumberUtils, ChaosMonkeySafeLeaderTest, TestJmxIntegration, FieldAnalysisRequestHandlerTest, SolrIndexConfigTest, SimplePostToolTest, TestStressRecovery, AliasIntegrationTest, HdfsBasicDistributedZkTest, TestComponentsName, BasicDistributedZk2Test, TestSimpleQParserPlugin, SuggestComponentTest, TestCustomSort, DirectUpdateHandlerTest, TestSchemaSimilarityResource, StandardRequestHandlerTest, ConvertedLegacyTest, IndexSchemaRuntimeFieldTest, TestSolrQueryParserDefaultOperatorResource, HdfsDirectoryTest, TestArbitraryIndexDir, HighlighterConfigTest, TestDynamicFieldCollectionResource, BasicZkTest, DebugComponentTest, DirectSolrSpellCheckerTest, SolrPluginUtilsTest, DistributedQueryComponentCustomSortTest, DocValuesMultiTest, HighlighterTest, CursorMarkTest, TestFastOutputStream, TestDistribDocBasedVersion, OpenCloseCoreStressTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed on J1 in 85.77s, 1 test, 1 error <<< FAILURES!

[...truncated 525 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 355 suites, 1546 tests, 1 error, 36 ignored (8 assumptions)

Total time: 48 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 9123 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9123/
Java: 32bit/jdk1.6.0_45 -server -XX:+UseConcMarkSweepGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:42929/collection2, http://127.0.0.1:53340/collection2, http://127.0.0.1:50101/collection2, http://127.0.0.1:51111/collection2, http://127.0.0.1:46849/collection2]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:42929/collection2, http://127.0.0.1:53340/collection2, http://127.0.0.1:50101/collection2, http://127.0.0.1:51111/collection2, http://127.0.0.1:46849/collection2]
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:352)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:410)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:143)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: no servers hosting shard: 
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:317)
	... 46 more


REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at __randomizedtesting.SeedInfo.seed([5D212428EDE8C1AB:5929ABDBFF4D2E8A]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:428)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:425)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)




Build Log:
[...truncated 9671 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> 465843 T1422 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 465847 T1422 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1390610471434
   [junit4]   2> 465847 T1422 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 465848 T1423 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 465948 T1422 oasc.ZkTestServer.run start zk server on port:54259
   [junit4]   2> 465949 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 465999 T1429 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1561668 name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 465999 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 466000 T1422 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 466008 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 466010 T1431 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f626a6 name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 466011 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 466011 T1422 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 466015 T1422 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 466019 T1422 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 466022 T1422 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 466026 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 466026 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 466032 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 466033 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 466037 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 466038 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 466042 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 466042 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 466046 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 466046 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 466050 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 466051 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 466055 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 466055 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 466059 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 466060 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 466067 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 466067 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 466071 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 466072 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 466076 T1422 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 466076 T1422 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 466152 T1422 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 466169 T1422 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42929
   [junit4]   2> 466169 T1422 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 466170 T1422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 466171 T1422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390610471668
   [junit4]   2> 466171 T1422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390610471668/'
   [junit4]   2> 466185 T1422 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390610471668/solr.xml
   [junit4]   2> 466234 T1422 oasc.CoreContainer.<init> New CoreContainer 26331522
   [junit4]   2> 466235 T1422 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390610471668/]
   [junit4]   2> 466242 T1422 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 466242 T1422 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 466243 T1422 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 466243 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 466244 T1422 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 466244 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 466245 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 466246 T1422 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 466246 T1422 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 466250 T1422 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 466250 T1422 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 466251 T1422 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 466252 T1422 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54259/solr
   [junit4]   2> 466252 T1422 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 466253 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 466256 T1442 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8bf1a name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 466257 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 466260 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 466262 T1444 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c7da7b name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 466263 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 466265 T1422 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 466271 T1422 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 466277 T1422 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 466280 T1422 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42929_
   [junit4]   2> 466282 T1422 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42929_
   [junit4]   2> 466287 T1422 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 466291 T1422 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 466301 T1422 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 466305 T1422 oasc.Overseer.start Overseer (id=91135047856029699-127.0.0.1:42929_-n_0000000000) starting
   [junit4]   2> 466313 T1422 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 466319 T1446 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 466321 T1422 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 466321 T1446 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 466326 T1422 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 466329 T1422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 466334 T1445 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 466340 T1447 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 466340 T1447 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 466343 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466343 T1447 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 466347 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 466348 T1445 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42929",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42929_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 466348 T1445 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 466349 T1445 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 466352 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466353 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 467344 T1447 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 467345 T1447 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390610471668/collection1
   [junit4]   2> 467345 T1447 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 467347 T1447 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 467347 T1447 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 467349 T1447 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390610471668/collection1/'
   [junit4]   2> 467350 T1447 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390610471668/collection1/lib/classes/' to classloader
   [junit4]   2> 467350 T1447 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390610471668/collection1/lib/README' to classloader
   [junit4]   2> 467393 T1447 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 467445 T1447 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 467447 T1447 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 467454 T1447 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 467873 T1447 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 467876 T1447 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 467877 T1447 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 467883 T1447 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 467920 T1447 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 467921 T1447 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1390610471668/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/control/data/
   [junit4]   2> 467921 T1447 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e73498
   [junit4]   2> 467922 T1447 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/control/data
   [junit4]   2> 467922 T1447 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/control/data/index/
   [junit4]   2> 467922 T1447 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 467922 T1447 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/control/data/index
   [junit4]   2> 467923 T1447 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=31.8955078125, floorSegmentMB=1.7353515625, forceMergeDeletesPctAllowed=3.0579649857860245, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3356732622114079
   [junit4]   2> 467923 T1447 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@d83f5e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1121aef),segFN=segments_1,generation=1}
   [junit4]   2> 467924 T1447 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 467926 T1447 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 467927 T1447 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 467927 T1447 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 467927 T1447 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 467927 T1447 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 467928 T1447 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 467928 T1447 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 467928 T1447 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 467928 T1447 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 467929 T1447 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 467929 T1447 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 467929 T1447 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 467930 T1447 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 467930 T1447 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 467931 T1447 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 467931 T1447 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 467940 T1447 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 467942 T1447 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 467943 T1447 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 467943 T1447 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=12, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=49.0546875, floorSegmentMB=1.125, forceMergeDeletesPctAllowed=9.39645753487684, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6107728522811995
   [junit4]   2> 467944 T1447 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@d83f5e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1121aef),segFN=segments_1,generation=1}
   [junit4]   2> 467944 T1447 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 467944 T1447 oass.SolrIndexSearcher.<init> Opening Searcher@1c6c0b9 main
   [junit4]   2> 467946 T1448 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c6c0b9 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 467948 T1447 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 467949 T1422 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 467949 T1422 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 467949 T1451 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42929 collection:control_collection shard:shard1
   [junit4]   2> 467950 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 467952 T1453 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9eca34 name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 467952 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 467953 T1451 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 467957 T1422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 467963 T1422 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 467966 T1451 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 467969 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 467970 T1451 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 467970 T1451 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C5054 name=collection1 org.apache.solr.core.SolrCore@1267641 url=http://127.0.0.1:42929/collection1 node=127.0.0.1:42929_ C5054_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:42929, core=collection1, node_name=127.0.0.1:42929_}
   [junit4]   2> 467970 T1451 C5054 P42929 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42929/collection1/
   [junit4]   2> 467971 T1451 C5054 P42929 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 467971 T1451 C5054 P42929 oasc.SyncStrategy.syncToMe http://127.0.0.1:42929/collection1/ has no replicas
   [junit4]   2> 467971 T1451 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42929/collection1/ shard1
   [junit4]   2> 467971 T1451 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 467972 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 467978 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 467980 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 467980 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 467990 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 467995 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468039 T1422 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 468040 T1422 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 468042 T1422 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49155
   [junit4]   2> 468042 T1422 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 468043 T1422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 468043 T1422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390610473551
   [junit4]   2> 468044 T1422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390610473551/'
   [junit4]   2> 468061 T1422 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390610473551/solr.xml
   [junit4]   2> 468098 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 468098 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 468114 T1422 oasc.CoreContainer.<init> New CoreContainer 18679784
   [junit4]   2> 468114 T1422 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390610473551/]
   [junit4]   2> 468116 T1422 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 468116 T1422 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 468116 T1422 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 468117 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 468117 T1422 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 468117 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 468118 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 468118 T1422 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 468119 T1422 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 468121 T1422 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 468121 T1422 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 468122 T1422 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 468122 T1422 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54259/solr
   [junit4]   2> 468123 T1422 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 468123 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 468125 T1464 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f24c69 name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 468125 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 468128 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 468130 T1466 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@150f4bc name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 468130 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 468137 T1422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 468144 T1451 oasc.ZkController.register We are http://127.0.0.1:42929/collection1/ and leader is http://127.0.0.1:42929/collection1/
   [junit4]   2> 468145 T1451 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42929
   [junit4]   2> 468145 T1451 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 468145 T1451 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 468145 T1451 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 468147 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468147 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468148 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468149 T1451 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 468152 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 468153 T1445 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42929",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42929_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 468156 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 468259 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 468259 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 468259 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 469151 T1422 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49155_
   [junit4]   2> 469154 T1422 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49155_
   [junit4]   2> 469159 T1444 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 469159 T1453 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 469160 T1466 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 469168 T1467 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 469168 T1467 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 469169 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 469170 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 469170 T1467 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 469170 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 469172 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 469173 T1445 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49155",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49155_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 469173 T1445 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 469173 T1445 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 469176 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 469280 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 469281 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 469281 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 470171 T1467 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 470172 T1467 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390610473551/collection1
   [junit4]   2> 470172 T1467 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 470173 T1467 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 470173 T1467 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 470175 T1467 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390610473551/collection1/'
   [junit4]   2> 470175 T1467 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390610473551/collection1/lib/classes/' to classloader
   [junit4]   2> 470176 T1467 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390610473551/collection1/lib/README' to classloader
   [junit4]   2> 470219 T1467 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 470275 T1467 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 470277 T1467 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 470284 T1467 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 470759 T1467 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 470762 T1467 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 470763 T1467 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 470767 T1467 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 470806 T1467 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 470806 T1467 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1390610473551/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty1/
   [junit4]   2> 470806 T1467 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e73498
   [junit4]   2> 470807 T1467 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty1
   [junit4]   2> 470807 T1467 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty1/index/
   [junit4]   2> 470808 T1467 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 470808 T1467 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty1/index
   [junit4]   2> 470809 T1467 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=31.8955078125, floorSegmentMB=1.7353515625, forceMergeDeletesPctAllowed=3.0579649857860245, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3356732622114079
   [junit4]   2> 470809 T1467 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@f478f3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1cff607),segFN=segments_1,generation=1}
   [junit4]   2> 470810 T1467 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 470814 T1467 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 470814 T1467 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 470814 T1467 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 470815 T1467 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 470815 T1467 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 470815 T1467 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 470815 T1467 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 470816 T1467 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 470816 T1467 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 470817 T1467 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 470817 T1467 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 470818 T1467 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 470818 T1467 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 470818 T1467 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 470819 T1467 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 470819 T1467 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 470821 T1467 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 470824 T1467 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 470824 T1467 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 470825 T1467 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=12, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=49.0546875, floorSegmentMB=1.125, forceMergeDeletesPctAllowed=9.39645753487684, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6107728522811995
   [junit4]   2> 470839 T1467 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@f478f3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1cff607),segFN=segments_1,generation=1}
   [junit4]   2> 470839 T1467 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 470839 T1467 oass.SolrIndexSearcher.<init> Opening Searcher@7fb653 main
   [junit4]   2> 470844 T1468 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7fb653 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 470846 T1467 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 470847 T1422 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 470847 T1422 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 470847 T1471 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49155 collection:collection1 shard:shard1
   [junit4]   2> 470849 T1471 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 470861 T1471 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 470865 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470865 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470865 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470867 T1471 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 470867 T1471 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C5055 name=collection1 org.apache.solr.core.SolrCore@148c2ee url=http://127.0.0.1:49155/collection1 node=127.0.0.1:49155_ C5055_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:49155, core=collection1, node_name=127.0.0.1:49155_}
   [junit4]   2> 470867 T1471 C5055 P49155 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49155/collection1/
   [junit4]   2> 470868 T1471 C5055 P49155 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 470868 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 470868 T1471 C5055 P49155 oasc.SyncStrategy.syncToMe http://127.0.0.1:49155/collection1/ has no replicas
   [junit4]   2> 470868 T1471 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49155/collection1/ shard1
   [junit4]   2> 470869 T1471 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 470874 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470876 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 470876 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 470877 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 470887 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470890 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 470894 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470924 T1422 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 470925 T1422 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 470927 T1422 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51111
   [junit4]   2> 470928 T1422 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 470928 T1422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 470928 T1422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390610476435
   [junit4]   2> 470929 T1422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390610476435/'
   [junit4]   2> 470947 T1422 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390610476435/solr.xml
   [junit4]   2> 470991 T1422 oasc.CoreContainer.<init> New CoreContainer 25688998
   [junit4]   2> 470992 T1422 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390610476435/]
   [junit4]   2> 470993 T1422 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 470993 T1422 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 470994 T1422 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 470994 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 470994 T1422 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 470995 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 470995 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 470996 T1422 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 470996 T1422 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 470997 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 470997 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 470998 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 470998 T1422 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 470999 T1422 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 470999 T1422 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 471000 T1422 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54259/solr
   [junit4]   2> 471000 T1422 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 471001 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 471002 T1482 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fddf46 name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 471003 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 471007 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 471009 T1484 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1119b41 name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 471009 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 471015 T1422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 471043 T1471 oasc.ZkController.register We are http://127.0.0.1:49155/collection1/ and leader is http://127.0.0.1:49155/collection1/
   [junit4]   2> 471043 T1471 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49155
   [junit4]   2> 471044 T1471 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 471044 T1471 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 471044 T1471 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 471046 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 471055 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 471056 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 471056 T1471 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 471060 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 471062 T1445 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49155",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49155_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 471066 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 471170 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 471170 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 471171 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 471170 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 472024 T1422 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51111_
   [junit4]   2> 472027 T1422 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51111_
   [junit4]   2> 472031 T1453 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 472032 T1484 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 472033 T1466 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 472033 T1444 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 472040 T1485 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 472040 T1485 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 472041 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 472042 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 472042 T1485 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 472042 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 472044 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 472045 T1445 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51111",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51111_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 472046 T1445 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 472046 T1445 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 472049 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 472152 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 472152 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 472152 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 472153 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473042 T1485 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 473043 T1485 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390610476435/collection1
   [junit4]   2> 473043 T1485 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 473044 T1485 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 473044 T1485 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 473046 T1485 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390610476435/collection1/'
   [junit4]   2> 473046 T1485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390610476435/collection1/lib/classes/' to classloader
   [junit4]   2> 473047 T1485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390610476435/collection1/lib/README' to classloader
   [junit4]   2> 473092 T1485 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 473145 T1485 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 473147 T1485 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 473155 T1485 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 473600 T1485 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 473603 T1485 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 473605 T1485 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 473609 T1485 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 473643 T1485 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 473643 T1485 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1390610476435/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty2/
   [junit4]   2> 473643 T1485 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e73498
   [junit4]   2> 473644 T1485 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty2
   [junit4]   2> 473644 T1485 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty2/index/
   [junit4]   2> 473645 T1485 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 473645 T1485 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1390610471434/jetty2/index
   [junit4]   2> 473646 T1485 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=31.8955078125, floorSegmentMB=1.7353515625, forceMergeDeletesPctAllowed=3.0579649857860245, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3356732622114079
   [junit4]   2> 473646 T1485 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@84b463 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@107c261),segFN=segments_1,generation=1}
   [junit4]   2> 473647 T1485 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 473650 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 473651 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 473651 T1485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 473651 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 473651 T1485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 473652 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 473652 T1485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 473652 T1485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 473653 T1485 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 473653 T1485 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 473654 T1485 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 473654 T1485 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 473655 T1485 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 473655 T1485 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 473655 T1485 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 473656 T1485 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 473658 T1485 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 473661 T1485 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 473661 T1485 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 473668 T1485 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=12, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=49.0546875, floorSegmentMB=1.125, forceMergeDeletesPctAllowed=9.39645753487684, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6107728522811995
   [junit4]   2> 473668 T1485 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@84b463 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@107c261),segFN=segments_1,generation=1}
   [junit4]   2> 473668 T1485 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 473669 T1485 oass.SolrIndexSearcher.<init> Opening Searcher@1f907ee main
   [junit4]   2> 473672 T1486 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f907ee main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 473674 T1485 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 473674 T1422 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 473675 T1422 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 473674 T1489 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51111 collection:collection1 shard:shard2
   [junit4]   2> 473677 T1489 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 473687 T1489 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 473689 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473690 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473690 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473691 T1489 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 473691 T1489 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C5056 name=collection1 org.apache.solr.core.SolrCore@1c50f5a url=http://127.0.0.1:51111/collection1 node=127.0.0.1:51111_ C5056_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51111, core=collection1, node_name=127.0.0.1:51111_}
   [junit4]   2> 473691 T1489 C5056 P51111 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51111/collection1/
   [junit4]   2> 473691 T1489 C5056 P51111 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 473692 T1489 C5056 P51111 oasc.SyncStrategy.syncToMe http://127.0.0.1:51111/collection1/ has no replicas
   [junit4]   2> 473692 T1489 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51111/collection1/ shard2
   [junit4]   2> 473692 T1489 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 473692 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 473699 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473701 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473701 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473701 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473701 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473712 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 473717 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473775 T1422 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 473776 T1422 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 473778 T1422 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53340
   [junit4]   2> 473778 T1422 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 473779 T1422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 473779 T1422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390610479263
   [junit4]   2> 473779 T1422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390610479263/'
   [junit4]   2> 473802 T1422 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390610479263/solr.xml
   [junit4]   2> 473819 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473820 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473819 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473820 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473848 T1422 oasc.CoreContainer.<init> New CoreContainer 117934
   [junit4]   2> 473848 T1422 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1390610479263/]
   [junit4]   2> 473849 T1422 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 473850 T1422 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 473850 T1422 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 473851 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 473851 T1422 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 473851 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 473852 T1422 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 473852 T1422 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 473853 T1422 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 473855 T1422 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 473855 T1422 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 473856 T1422 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 473856 T1422 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54259/solr
   [junit4]   2> 473857 T1422 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 473857 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 473860 T1500 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24c700 name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 473860 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 473861 T1489 oasc.ZkController.register We are http://127.0.0.1:51111/collection1/ and leader is http://127.0.0.1:51111/collection1/
   [junit4]   2> 473861 T1489 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51111
   [junit4]   2> 473861 T1489 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 473862 T1489 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 473862 T1489 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 473864 T1422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 473864 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473865 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473865 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473866 T1502 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b4b78 name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 473866 T1422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 473866 T1489 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 473869 T1445 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 473871 T1445 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51111",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51111_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 473882 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473882 T1422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 473986 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473986 T1484 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473986 T1444 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473986 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 473986 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 474888 T1422 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53340_
   [junit4]   2> 474889 T1422 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53340_
   [junit4]   2> 474895 T1484 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 474895 T1453 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 474895 T1502 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 474896 T1466 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 474896 T1444 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 474904 T1503 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 474905 T1503 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 474906 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474906 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474906 T1444 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConne

[...truncated too long message...]

.cloud.ConnectionManager@6fac01 name:ZooKeeperConnection Watcher:127.0.0.1:55874/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1141166 T4488 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1141167 T4488 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1141170 T4919 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b4bb35 name:ZooKeeperConnection Watcher:127.0.0.1:55874 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1141170 T4488 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1141201 T4488 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1141202 T4921 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@330fc4 name:ZooKeeperConnection Watcher:127.0.0.1:55874 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1141202 T4488 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1141203 T4488 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1141206 T4488 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1141209 T4488 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1141212 T4488 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1141215 T4488 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1141220 T4488 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1141225 T4488 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1141226 T4923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bdef7 name:ZooKeeperConnection Watcher:127.0.0.1:55874/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1141226 T4488 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1141227 T4488 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1141231 T4488 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1141238 T4488 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1141241 T4488 oasc.Overseer.start Overseer (id=91135092108099587-127.0.0.1:55874_solr-n_0000000000) starting
   [junit4]   2> 1141244 T4488 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1141250 T4488 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1141253 T4925 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1141255 T4925 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1141258 T4924 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1141259 T4924 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 1141259 T4924 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1141260 T4917 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1141265 T4924 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1141267 T4917 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1141272 T4924 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1141277 T4924 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1141279 T4924 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1141285 T4923 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1141286 T4917 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1141361 T4488 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55874 55874
   [junit4]   2> 1141644 T4488 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1141646 T4488 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 1141646 T4488 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 1144646 T4488 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=DefaultSimilarity, locale=en_NZ, timezone=America/Denver
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=8,threads=1,free=102476984,total=530186240
   [junit4]   2> NOTE: All tests run in this JVM: [XsltUpdateRequestHandlerTest, TestMultiCoreConfBootstrap, TestDefaultSearchFieldResource, HdfsSyncSliceTest, CachingDirectoryFactoryTest, TestRTGBase, TestPhraseSuggestions, OpenExchangeRatesOrgProviderTest, StatsComponentTest, TestIndexSearcher, TestUpdate, SpellingQueryConverterTest, TestComponentsName, DirectSolrSpellCheckerTest, SuggestComponentTest, ZkNodePropsTest, TestFastOutputStream, TestNoOpRegenerator, SignatureUpdateProcessorFactoryTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, CoreMergeIndexesAdminHandlerTest, TestFreeTextSuggestions, HdfsChaosMonkeySafeLeaderTest, DocValuesMultiTest, NumericFieldsTest, DistanceFunctionTest, TestStressReorder, DistributedQueryComponentCustomSortTest, SpellCheckCollatorTest, DateFieldTest, ChaosMonkeyNothingIsSafeTest, TestDynamicFieldResource, DefaultValueUpdateProcessorTest, XmlUpdateRequestHandlerTest, TestDistribDocBasedVersion, SolrCmdDistributorTest, TestSolrCoreProperties, TestCodecSupport, TestDFRSimilarityFactory, CursorPagingTest, TestIndexingPerformance, AddBlockUpdateTest, TestAnalyzedSuggestions, NoCacheHeaderTest, PathHierarchyTokenizerFactoryTest, RAMDirectoryFactoryTest, FullSolrCloudDistribCmdsTest, TestImplicitCoreProperties, ConvertedLegacyTest, TermVectorComponentDistributedTest, SortByFunctionTest, DebugComponentTest, TestNRTOpen, DistributedSuggestComponentTest, HdfsCollectionsAPIDistributedZkTest, TestCursorMarkWithoutUniqueKey, DirectSolrConnectionTest, TestArbitraryIndexDir, StandardRequestHandlerTest, TestSchemaResource, TestHashPartitioner, TestSerializedLuceneMatchVersion, TestIBSimilarityFactory, TestSimpleQParserPlugin, HdfsBasicDistributedZkTest, TestFileDictionaryLookup, MultiTermTest, TestDistributedSearch, TestStressRecovery, TestSchemaSimilarityResource, TestFieldTypeResource, ZkSolrClientTest, UniqFieldsUpdateProcessorFactoryTest, BasicDistributedZk2Test, TestCollationField, TestRangeQuery, HighlighterConfigTest, TestLazyCores, ShardSplitTest, SolrCoreCheckLockOnStartupTest, PrimUtilsTest, TestFoldingMultitermQuery, TestJmxIntegration, CursorMarkTest, TestQuerySenderNoQuery, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, OverseerTest]
   [junit4] Completed on J1 in 30.83s, 8 tests, 1 error <<< FAILURES!

[...truncated 549 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 355 suites, 1546 tests, 2 errors, 35 ignored (7 assumptions)

Total time: 62 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b03) - Build # 9122 - Still Failing!

Posted by Steve Rowe <sa...@gmail.com>.
check-licenses:
     [echo] License check under: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene
 [licenses] CHECKSUM FAILED for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/benchmark/lib/spatial4j-0.4.jar (expected: "28b666145773098aeeadd033f3dd6c5b2871c208" was: "3fb96c751e83cd815b68e5f53d058f2cb8a9ecaa")
 [licenses] CHECKSUM FAILED for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/spatial/lib/spatial4j-0.4.jar (expected: "28b666145773098aeeadd033f3dd6c5b2871c208" was: "3fb96c751e83cd815b68e5f53d058f2cb8a9ecaa")
 [licenses] Scanned 34 JAR file(s) for licenses (in 0.53s.), 2 error(s).


On Jan 24, 2014, at 6:13 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9122/
> Java: 64bit/jdk1.7.0_60-ea-b03 -XX:+UseCompressedOops -XX:+UseParallelGC
> 
> All tests passed
> 
> Build Log:
> [...truncated 20296 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:64: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:162: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.
> 
> Total time: 47 minutes 26 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0_60-ea-b03 -XX:+UseCompressedOops -XX:+UseParallelGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org