You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/05/16 00:48:17 UTC
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still
Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/1621/
3 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:20022/f returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:20022/f returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([59CFFC3A247B8528:D82972225324E514]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
FAILED: junit.framework.TestSuite.org.apache.solr.update.processor.RegexBoostProcessorTest
Error Message:
Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 51,150,032 bytes (threshold is 10,485,760). Field reference sizes (counted individually): - 51,880,376 bytes, protected static org.apache.solr.servlet.SolrRequestParsers org.apache.solr.update.processor.RegexBoostProcessorTest._parser - 5,256 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessorFactory org.apache.solr.update.processor.RegexBoostProcessorTest.factory - 4,568 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessor org.apache.solr.update.processor.RegexBoostProcessorTest.reProcessor - 856 bytes, protected static org.apache.solr.common.params.ModifiableSolrParams org.apache.solr.update.processor.RegexBoostProcessorTest.parameters - 296 bytes, protected static java.lang.String org.apache.solr.SolrTestCaseJ4.testSolrHome - 272 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules - 136 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp - 72 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName
Stack Trace:
junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 51,150,032 bytes (threshold is 10,485,760). Field reference sizes (counted individually):
- 51,880,376 bytes, protected static org.apache.solr.servlet.SolrRequestParsers org.apache.solr.update.processor.RegexBoostProcessorTest._parser
- 5,256 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessorFactory org.apache.solr.update.processor.RegexBoostProcessorTest.factory
- 4,568 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessor org.apache.solr.update.processor.RegexBoostProcessorTest.reProcessor
- 856 bytes, protected static org.apache.solr.common.params.ModifiableSolrParams org.apache.solr.update.processor.RegexBoostProcessorTest.parameters
- 296 bytes, protected static java.lang.String org.apache.solr.SolrTestCaseJ4.testSolrHome
- 272 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules
- 136 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp
- 72 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName
at __randomizedtesting.SeedInfo.seed([59CFFC3A247B8528]:0)
at com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.java:127)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:19139/g/d/collection1lastClient and got 5 from http://127.0.0.1:57375/g/d/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:19139/g/d/collection1lastClient and got 5 from http://127.0.0.1:57375/g/d/collection1
at __randomizedtesting.SeedInfo.seed([59CFFC3A247B8528:D82972225324E514]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
Build Log:
[...truncated 8754 lines...]
[junit4:junit4] Suite: org.apache.solr.update.processor.RegexBoostProcessorTest
[junit4:junit4] 2> 411300 T1360 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373
[junit4:junit4] 2> 411301 T1360 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4] 2> 411302 T1360 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4] 2> 411303 T1360 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 411388 T1360 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 411466 T1360 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 411467 T1360 oass.IndexSchema.readSchema Reading Solr Schema from schema12.xml
[junit4:junit4] 2> 411474 T1360 oass.IndexSchema.readSchema [null] Schema name=test
[junit4:junit4] 2> 412044 T1360 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 412048 T1360 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 412051 T1360 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (2)
[junit4:junit4] 2> 412063 T1360 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 412068 T1360 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 412072 T1360 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 412073 T1360 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4] 2> 412073 T1360 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4] 2> 412087 T1360 oasc.CoreContainer.<init> New CoreContainer 1629024597
[junit4:junit4] 2> 412131 T1360 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 412131 T1360 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373/
[junit4:junit4] 2> 412132 T1360 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f78dc08
[junit4:junit4] 2> 412132 T1360 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 412133 T1360 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373
[junit4:junit4] 2> 412133 T1360 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373/index/
[junit4:junit4] 2> 412133 T1360 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 412134 T1360 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373/index
[junit4:junit4] 2> 412136 T1360 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=RateLimitedDirectoryWrapper(MockDirWrapper(org.apache.lucene.store.RAMDirectory@7c23285c lockFactory=org.apache.lucene.store.NativeFSLockFactory@107b8627)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 412136 T1360 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 412137 T1360 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4] 2> 412138 T1360 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 412139 T1360 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4] 2> 412139 T1360 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 412139 T1360 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4] 2> 412140 T1360 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 412140 T1360 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 412140 T1360 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 412141 T1360 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 412141 T1360 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 412142 T1360 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4] 2> 412142 T1360 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4] 2> 412142 T1360 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 412143 T1360 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4] 2> 412143 T1360 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4] 2> 412143 T1360 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4] 2> 412144 T1360 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 412144 T1360 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 412145 T1360 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 412145 T1360 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 412145 T1360 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 412146 T1360 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 412146 T1360 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 412147 T1360 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 412147 T1360 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 412147 T1360 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4] 2> 412148 T1360 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4] 2> 412153 T1360 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 412156 T1360 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4] 2> 412158 T1360 oasc.SolrCore.initDeprecatedSupport WARN adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-REPLICATION2.XML, SCHEMA-MINIMAL.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, TRIVIAL.UPDATEPROCESSOR1.JS, BAD-SCHEMA-DOCVALUES-NOT-REQUIRED-NO-DEFAULT.XML, SOLRCONFIG-CACHING.XML, BAD-SOLRCONFIG-UNEXPECTED-SCHEMA-ATTRIBUTE.XML, SCHEMA-POSTINGSHIGHLIGHT.XML, REGEX-BOOST-PROCESSOR-TEST.TXT, SOLRCONFIG-REPEATER.XML, CURRENCY.XML, BAD-SCHEMA-CURRENCY-FT-BOGUS-CODE-IN-XML.XML, SOLRCONFIG-MERGEPOLICY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, SOLRCONFIG-COMPONENTS-NAME.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, ELEVATE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, TRIVIAL.UPDATEPROCESSOR0.JS, SCHEMA-SWEETSPOT.XML, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, BAD-SCHEMA-MISPLACED-ASTERISK-COPYFIELD-SOURCE-SHOULD-FAIL-TEST.XML, DA_UTF8.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, BAD-SCHEMA-NON-GLOB-COPYFIELD-SOURCE-MATCHING-NOTHING-SHOULD-FAIL-TEST.XML, CONDITIONAL.UPDATEPROCESSOR.JS, SCHEMA-TINY.XML, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SOLRCONFIG-ENABLEPLUGIN.XML, HYPHENATION.DTD, BAD-SOLRCONFIG-SCHEMA-MUTABLE-BUT-NOT-MANAGED.XML, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, SCHEMA-ID-AND-VERSION-FIELDS-ONLY.XML, HUNSPELL-TEST.AFF, BAD-SCHEMA-CURRENCY-FT-OER-NORATES.XML, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, SOLRCONFIG-TLOG-MUTABLE-MANAGED-SCHEMA.XML, SCHEMA-DOCVALUES.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, BAD-SCHEMA-BOGUS-ANALYSIS-PARAMETERS.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, SCHEMA-SPATIAL.XML, OPEN-EXCHANGE-RATES.JSON, STOPWITHBOM.TXT, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-SPELLCHECKER.XML, SOLRCONFIG-POSTINGSHIGHLIGHT.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, SOLRCONFIG-WARMER.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, BAD-SCHEMA-MULTIPLE-ASTERISK-COPYFIELD-SOURCE-SHOULD-FAIL-TEST.XML, SCHEMA-BEHAVIOR.XML, SYNONYMS.TXT, SOLRCONFIG-MINIMAL.XML, SCHEMA-DOCVALUESMULTI.XML, BAD-SCHEMA-UNSUPPORTED-DOCVALUES.XML, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, BAD-SCHEMA-SWEETSPOT-PARTIAL-HYPERBOLIC.XML, SOLRCONFIG-SOLR-749.XML, SCHEMA-EFF.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SCHEMA-ONE-FIELD-NO-DYNAMIC-FIELD.XML, SOLRCONFIG-ELEVATE.XML, BAD-SCHEMA-SWEETSPOT-PARTIAL-NORMS.XML, BAD-SCHEMA-SWEETSPOT-PARTIAL-BASELINE.XML, SCHEMA-FOLDING.XML, STOPWORDS.TXT, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-BOGUS-FIELD-PARAMETERS.XML, FUZZYSUGGEST.TXT, BAD-SOLRCONFIG-MANAGED-SCHEMA-NAMED-SCHEMA.XML.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SCHEMA-REST.XML, SOLRCONFIG-TLOG-MANAGED-SCHEMA.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SOLRCONFIG-MANAGED-SCHEMA.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, BAD-SCHEMA-MISPLACED-ASTERISK-COPYFIELD-DEST-SHOULD-FAIL-TEST.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, SCHEMA-CLASS-NAME-SHORTENING-ON-SERIALIZATION.XML, BAD-MP-SOLRCONFIG.XML, FRENCHARTICLES.TXT, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, BAD-SCHEMA-MULTIPLE-ASTERISK-COPYFIELD-DEST-SHOULD-FAIL-TEST.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, SCHEMA-SYNONYM-TOKENIZER.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-REST-LUCENE-MATCH-VERSION.XML, SCHEMA-DOCVALUESFACETING.XML, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-MUTABLE-MANAGED-SCHEMA.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, BAD-SCHEMA-SWEETSPOT-BOTH-TF.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, BAD-CURRENCY.XML, BAD-SCHEMA-CURRENCY-FT-BOGUS-DEFAULT-CODE.XML, SOLRCONFIG_PERF.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4] 2> 412160 T1360 oass.SolrIndexSearcher.<init> Opening Searcher@41f0b28 main
[junit4:junit4] 2> 412165 T1360 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 412165 T1360 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 412166 T1360 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4] 2> 412172 T1360 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4] 2> 412228 T1361 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@41f0b28 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 412231 T1360 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 412237 T1360 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4] 2> 412245 T1360 oas.SolrTestCaseJ4.setUp ###Starting testDeboostOld
[junit4:junit4] 2> 412246 T1360 oas.SolrTestCaseJ4.tearDown ###Ending testDeboostOld
[junit4:junit4] 2> 412249 T1360 oas.SolrTestCaseJ4.setUp ###Starting testNoBoost
[junit4:junit4] 2> 412250 T1360 oas.SolrTestCaseJ4.tearDown ###Ending testNoBoost
[junit4:junit4] 2> 412253 T1360 oas.SolrTestCaseJ4.setUp ###Starting testTwoRules
[junit4:junit4] 2> 412253 T1360 oas.SolrTestCaseJ4.tearDown ###Ending testTwoRules
[junit4:junit4] 2> 412257 T1360 oas.SolrTestCaseJ4.setUp ###Starting testBoostGood
[junit4:junit4] 2> 412257 T1360 oas.SolrTestCaseJ4.tearDown ###Ending testBoostGood
[junit4:junit4] 2> 412258 T1360 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 412258 T1360 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1629024597
[junit4:junit4] 2> 412258 T1360 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@2c8c3540
[junit4:junit4] 2> 412265 T1360 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 412266 T1360 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 412266 T1360 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 412266 T1360 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 412266 T1360 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 412267 T1360 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 412267 T1360 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373/index;done=false>>]
[junit4:junit4] 2> 412267 T1360 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373/index
[junit4:junit4] 2> 412267 T1360 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373;done=false>>]
[junit4:junit4] 2> 412267 T1360 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368656409373
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=ga_IE, timezone=Pacific/Pago_Pago
[junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=225010856,total=478281728
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestQueryUtils, TestSearchPerf, QueryResultKeyTest, TestXIncludeConfig, DocumentBuilderTest, TestFuzzyAnalyzedSuggestions, LegacyHTMLStripCharFilterTest, ClusterStateUpdateTest, TestReversedWildcardFilterFactory, TestPluginEnable, BasicDistributedZkTest, TestSolrIndexConfig, MinimalSchemaTest, TestDynamicFieldResource, PrimitiveFieldTypeTest, TestPhraseSuggestions, TestReplicationHandler, ShowFileRequestHandlerTest, TestLRUCache, FieldMutatingUpdateProcessorTest, AliasIntegrationTest, RegexBoostProcessorTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RegexBoostProcessorTest -Dtests.seed=59CFFC3A247B8528 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ga_IE -Dtests.timezone=Pacific/Pago_Pago -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | RegexBoostProcessorTest (suite) <<<
[junit4:junit4] > Throwable #1: junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 51,150,032 bytes (threshold is 10,485,760). Field reference sizes (counted individually):
[junit4:junit4] > - 51,880,376 bytes, protected static org.apache.solr.servlet.SolrRequestParsers org.apache.solr.update.processor.RegexBoostProcessorTest._parser
[junit4:junit4] > - 5,256 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessorFactory org.apache.solr.update.processor.RegexBoostProcessorTest.factory
[junit4:junit4] > - 4,568 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessor org.apache.solr.update.processor.RegexBoostProcessorTest.reProcessor
[junit4:junit4] > - 856 bytes, protected static org.apache.solr.common.params.ModifiableSolrParams org.apache.solr.update.processor.RegexBoostProcessorTest.parameters
[junit4:junit4] > - 296 bytes, protected static java.lang.String org.apache.solr.SolrTestCaseJ4.testSolrHome
[junit4:junit4] > - 272 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules
[junit4:junit4] > - 136 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp
[junit4:junit4] > - 72 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([59CFFC3A247B8528]:0)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:679)
[junit4:junit4] Completed on J0 in 2.79s, 4 tests, 1 failure <<< FAILURES!
[...truncated 50 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4] 2> 357374 T740 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /f/
[junit4:junit4] 2> 357382 T740 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyShardSplitTest-1368656355583
[junit4:junit4] 2> 357383 T740 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 357384 T741 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 357485 T740 oasc.ZkTestServer.run start zk server on port:20015
[junit4:junit4] 2> 357485 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 357490 T747 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22d3ac67 name:ZooKeeperConnection Watcher:127.0.0.1:20015 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 357490 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 357491 T740 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 357503 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 357506 T749 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a1f2bee name:ZooKeeperConnection Watcher:127.0.0.1:20015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 357506 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 357507 T740 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 357517 T740 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 357521 T740 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 357523 T740 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 357532 T740 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 357532 T740 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 357546 T740 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 357546 T740 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 357650 T740 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 357650 T740 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 357654 T740 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 357654 T740 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 357658 T740 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 357658 T740 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 357661 T740 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 357662 T740 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 357666 T740 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 357666 T740 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 357669 T740 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 357670 T740 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 357673 T740 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 357673 T740 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 357820 T740 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 357823 T740 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:20018
[junit4:junit4] 2> 357823 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 357824 T740 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 357824 T740 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368656355885
[junit4:junit4] 2> 357824 T740 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368656355885/solr.xml
[junit4:junit4] 2> 357825 T740 oasc.CoreContainer.<init> New CoreContainer 191968531
[junit4:junit4] 2> 357825 T740 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368656355885/'
[junit4:junit4] 2> 357826 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368656355885/'
[junit4:junit4] 2> 357893 T740 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 357893 T740 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 357894 T740 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 357894 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 357894 T740 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 357895 T740 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 357895 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 357895 T740 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 357896 T740 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 357896 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 357907 T740 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 357908 T740 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:20015/solr
[junit4:junit4] 2> 357908 T740 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 357909 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 357911 T760 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7afc920e name:ZooKeeperConnection Watcher:127.0.0.1:20015 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 357912 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 357914 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 357923 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 357929 T762 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2be2ef3e name:ZooKeeperConnection Watcher:127.0.0.1:20015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 357930 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 357932 T740 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 357941 T740 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 357949 T740 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 357951 T740 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:20018_f
[junit4:junit4] 2> 357953 T740 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:20018_f
[junit4:junit4] 2> 357963 T740 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 357977 T740 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 357980 T740 oasc.Overseer.start Overseer (id=89696262919815171-127.0.0.1:20018_f-n_0000000000) starting
[junit4:junit4] 2> 357992 T740 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 358004 T764 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 358005 T740 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 358008 T740 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 358038 T740 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 358042 T763 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 358046 T765 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368656355885/collection1
[junit4:junit4] 2> 358046 T765 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 358047 T765 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 358047 T765 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 358049 T765 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368656355885/collection1/'
[junit4:junit4] 2> 358050 T765 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368656355885/collection1/lib/README' to classloader
[junit4:junit4] 2> 358051 T765 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368656355885/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 358099 T765 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 358157 T765 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 358259 T765 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 358265 T765 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 358830 T765 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 358834 T765 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 358837 T765 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 358842 T765 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 358874 T765 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 358875 T765 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368656355885/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/control/data/
[junit4:junit4] 2> 358875 T765 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f702c48
[junit4:junit4] 2> 358876 T765 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 358876 T765 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/control/data
[junit4:junit4] 2> 358877 T765 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/control/data/index/
[junit4:junit4] 2> 358877 T765 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 358878 T765 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/control/data/index
[junit4:junit4] 2> 358882 T765 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2c786e17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1017e84b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 358882 T765 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 358885 T765 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 358885 T765 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 358886 T765 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 358886 T765 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 358887 T765 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 358887 T765 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 358888 T765 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 358888 T765 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 358889 T765 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 358896 T765 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 358902 T765 oass.SolrIndexSearcher.<init> Opening Searcher@66eeca3a main
[junit4:junit4] 2> 358903 T765 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/control/data/tlog
[junit4:junit4] 2> 358904 T765 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 358904 T765 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 358908 T766 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@66eeca3a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 358910 T765 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 358910 T765 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 359546 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 359547 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20018_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20018/f"}
[junit4:junit4] 2> 359547 T763 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 359547 T763 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 359608 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 359913 T765 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 359913 T765 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:20018/f collection:control_collection shard:shard1
[junit4:junit4] 2> 359914 T765 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 359937 T765 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 359949 T765 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 359949 T765 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 359949 T765 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:20018/f/collection1/
[junit4:junit4] 2> 359950 T765 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 359950 T765 oasc.SyncStrategy.syncToMe http://127.0.0.1:20018/f/collection1/ has no replicas
[junit4:junit4] 2> 359950 T765 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:20018/f/collection1/
[junit4:junit4] 2> 359951 T765 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 361112 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 361134 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 361181 T765 oasc.ZkController.register We are http://127.0.0.1:20018/f/collection1/ and leader is http://127.0.0.1:20018/f/collection1/
[junit4:junit4] 2> 361181 T765 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:20018/f
[junit4:junit4] 2> 361181 T765 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 361182 T765 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 361182 T765 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 361184 T765 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 361185 T740 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 361186 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 361186 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 361201 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 361203 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 361205 T769 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e999262 name:ZooKeeperConnection Watcher:127.0.0.1:20015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 361205 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 361207 T740 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 361210 T740 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 361415 T740 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 361418 T740 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:20022
[junit4:junit4] 2> 361419 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 361419 T740 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 361420 T740 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368656359410
[junit4:junit4] 2> 361420 T740 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368656359410/solr.xml
[junit4:junit4] 2> 361421 T740 oasc.CoreContainer.<init> New CoreContainer 1975445244
[junit4:junit4] 2> 361421 T740 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368656359410/'
[junit4:junit4] 2> 361422 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368656359410/'
[junit4:junit4] 2> 361521 T740 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 361521 T740 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 361522 T740 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 361522 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 361523 T740 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 361523 T740 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 361524 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 361525 T740 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 361525 T740 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 361526 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 361540 T740 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 361541 T740 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:20015/solr
[junit4:junit4] 2> 361542 T740 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 361543 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 361546 T780 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f31797d name:ZooKeeperConnection Watcher:127.0.0.1:20015 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 361547 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 361563 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 361574 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 361576 T782 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55bce763 name:ZooKeeperConnection Watcher:127.0.0.1:20015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 361576 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 361587 T740 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 362590 T740 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:20022_f
[junit4:junit4] 2> 362592 T740 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:20022_f
[junit4:junit4] 2> 362596 T769 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 362596 T782 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 362596 T762 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 362597 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 362618 T783 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368656359410/collection1
[junit4:junit4] 2> 362618 T783 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 362619 T783 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 362619 T783 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 362621 T783 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368656359410/collection1/'
[junit4:junit4] 2> 362622 T783 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368656359410/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 362622 T783 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368656359410/collection1/lib/README' to classloader
[junit4:junit4] 2> 362639 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 362640 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:20018_f_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20018_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20018/f"}
[junit4:junit4] 2> 362650 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 362650 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 362650 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 362672 T783 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 362729 T783 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 362830 T783 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 362836 T783 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 363394 T783 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 363398 T783 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 363401 T783 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 363406 T783 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 363438 T783 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 363439 T783 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368656359410/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty1/
[junit4:junit4] 2> 363439 T783 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f702c48
[junit4:junit4] 2> 363440 T783 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 363441 T783 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty1
[junit4:junit4] 2> 363441 T783 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty1/index/
[junit4:junit4] 2> 363441 T783 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 363442 T783 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty1/index
[junit4:junit4] 2> 363445 T783 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6c452fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e2da07a),segFN=segments_1,generation=1}
[junit4:junit4] 2> 363446 T783 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 363449 T783 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 363449 T783 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 363450 T783 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 363451 T783 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 363451 T783 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 363452 T783 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 363452 T783 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 363453 T783 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 363453 T783 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 363461 T783 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 363467 T783 oass.SolrIndexSearcher.<init> Opening Searcher@37ba3e33 main
[junit4:junit4] 2> 363467 T783 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty1/tlog
[junit4:junit4] 2> 363468 T783 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 363469 T783 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 363473 T784 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37ba3e33 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 363476 T783 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 363476 T783 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 364156 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 364156 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20022_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20022/f"}
[junit4:junit4] 2> 364157 T763 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 364157 T763 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 364172 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 364172 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 364172 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 364479 T783 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 364479 T783 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:20022/f collection:collection1 shard:shard1
[junit4:junit4] 2> 364480 T783 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 364589 T783 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 364592 T783 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 364593 T783 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 364593 T783 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:20022/f/collection1/
[junit4:junit4] 2> 364593 T783 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 364594 T783 oasc.SyncStrategy.syncToMe http://127.0.0.1:20022/f/collection1/ has no replicas
[junit4:junit4] 2> 364594 T783 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:20022/f/collection1/
[junit4:junit4] 2> 364594 T783 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 365678 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 365702 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 365702 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 365702 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 365711 T783 oasc.ZkController.register We are http://127.0.0.1:20022/f/collection1/ and leader is http://127.0.0.1:20022/f/collection1/
[junit4:junit4] 2> 365711 T783 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:20022/f
[junit4:junit4] 2> 365711 T783 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 365712 T783 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 365712 T783 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 365714 T783 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 365715 T740 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 365716 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 365716 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 365936 T740 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 365939 T740 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:20025
[junit4:junit4] 2> 365940 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 365941 T740 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 365941 T740 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368656363931
[junit4:junit4] 2> 365942 T740 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368656363931/solr.xml
[junit4:junit4] 2> 365942 T740 oasc.CoreContainer.<init> New CoreContainer 580288398
[junit4:junit4] 2> 365943 T740 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368656363931/'
[junit4:junit4] 2> 365943 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368656363931/'
[junit4:junit4] 2> 366041 T740 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 366042 T740 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 366042 T740 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 366043 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 366043 T740 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 366044 T740 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 366044 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 366045 T740 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 366045 T740 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 366046 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 366063 T740 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 366063 T740 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:20015/solr
[junit4:junit4] 2> 366064 T740 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 366065 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 366068 T796 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a505786 name:ZooKeeperConnection Watcher:127.0.0.1:20015 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 366069 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 366085 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 366098 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 366101 T798 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@496833a4 name:ZooKeeperConnection Watcher:127.0.0.1:20015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 366101 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 366114 T740 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 367118 T740 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:20025_f
[junit4:junit4] 2> 367121 T740 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:20025_f
[junit4:junit4] 2> 367124 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 367124 T798 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367124 T762 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367125 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 367125 T769 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367125 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 367126 T782 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367142 T799 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368656363931/collection1
[junit4:junit4] 2> 367142 T799 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 367143 T799 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 367144 T799 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 367145 T799 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368656363931/collection1/'
[junit4:junit4] 2> 367146 T799 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368656363931/collection1/lib/README' to classloader
[junit4:junit4] 2> 367147 T799 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368656363931/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 367199 T799 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 367215 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 367216 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:20022_f_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20022_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20022/f"}
[junit4:junit4] 2> 367229 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 367229 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 367229 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 367229 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 367258 T799 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 367359 T799 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 367365 T799 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 367917 T799 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 367921 T799 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 367924 T799 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 367929 T799 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 367957 T799 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 367958 T799 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368656363931/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty2/
[junit4:junit4] 2> 367958 T799 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f702c48
[junit4:junit4] 2> 367959 T799 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 367960 T799 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty2
[junit4:junit4] 2> 367960 T799 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty2/index/
[junit4:junit4] 2> 367960 T799 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 367961 T799 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty2/index
[junit4:junit4] 2> 367964 T799 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e205b65 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f3ea022),segFN=segments_1,generation=1}
[junit4:junit4] 2> 367964 T799 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 367967 T799 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 367968 T799 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 367968 T799 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 367969 T799 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 367970 T799 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 367970 T799 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 367970 T799 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 367971 T799 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 367972 T799 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 367979 T799 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 367985 T799 oass.SolrIndexSearcher.<init> Opening Searcher@7f9fbe63 main
[junit4:junit4] 2> 367985 T799 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty2/tlog
[junit4:junit4] 2> 367986 T799 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 367986 T799 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 367991 T800 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7f9fbe63 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 367993 T799 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 367993 T799 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 368734 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 368735 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20025_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20025/f"}
[junit4:junit4] 2> 368736 T763 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 368736 T763 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 368750 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368750 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368750 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368750 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368996 T799 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 368996 T799 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:20025/f collection:collection1 shard:shard2
[junit4:junit4] 2> 368997 T799 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 369014 T799 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 369024 T799 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 369024 T799 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 369024 T799 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:20025/f/collection1/
[junit4:junit4] 2> 369025 T799 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 369025 T799 oasc.SyncStrategy.syncToMe http://127.0.0.1:20025/f/collection1/ has no replicas
[junit4:junit4] 2> 369025 T799 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:20025/f/collection1/
[junit4:junit4] 2> 369026 T799 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 370256 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 370303 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370303 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370303 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370303 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370316 T799 oasc.ZkController.register We are http://127.0.0.1:20025/f/collection1/ and leader is http://127.0.0.1:20025/f/collection1/
[junit4:junit4] 2> 370316 T799 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:20025/f
[junit4:junit4] 2> 370316 T799 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 370317 T799 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 370317 T799 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 370319 T799 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 370320 T740 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 370321 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 370321 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 370540 T740 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 370543 T740 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:20028
[junit4:junit4] 2> 370544 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 370545 T740 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 370545 T740 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368656368535
[junit4:junit4] 2> 370546 T740 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368656368535/solr.xml
[junit4:junit4] 2> 370546 T740 oasc.CoreContainer.<init> New CoreContainer 855214586
[junit4:junit4] 2> 370547 T740 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368656368535/'
[junit4:junit4] 2> 370547 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368656368535/'
[junit4:junit4] 2> 370645 T740 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 370646 T740 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 370646 T740 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 370647 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 370647 T740 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 370648 T740 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 370648 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 370649 T740 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 370649 T740 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 370650 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 370663 T740 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 370664 T740 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:20015/solr
[junit4:junit4] 2> 370665 T740 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 370666 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 370669 T812 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3de0a45 name:ZooKeeperConnection Watcher:127.0.0.1:20015 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 370669 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 370687 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 370696 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 370698 T814 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a6e2820 name:ZooKeeperConnection Watcher:127.0.0.1:20015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 370699 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 370709 T740 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 371713 T740 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:20028_f
[junit4:junit4] 2> 371740 T740 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:20028_f
[junit4:junit4] 2> 371743 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 371744 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 371744 T762 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 371744 T814 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 371744 T769 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 371745 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371746 T798 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 371746 T782 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 371745 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371752 T815 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368656368535/collection1
[junit4:junit4] 2> 371753 T815 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 371753 T815 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 371754 T815 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 371755 T815 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368656368535/collection1/'
[junit4:junit4] 2> 371757 T815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368656368535/collection1/lib/README' to classloader
[junit4:junit4] 2> 371757 T815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368656368535/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 371806 T815 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 371809 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 371810 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:20025_f_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20025_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20025/f"}
[junit4:junit4] 2> 371824 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371824 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371824 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371824 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371824 T814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371865 T815 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 371966 T815 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 371972 T815 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 372514 T815 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 372517 T815 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 372519 T815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 372523 T815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 372544 T815 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 372544 T815 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368656368535/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/
[junit4:junit4] 2> 372544 T815 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f702c48
[junit4:junit4] 2> 372545 T815 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 372545 T815 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3
[junit4:junit4] 2> 372545 T815 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/index/
[junit4:junit4] 2> 372546 T815 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 372546 T815 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/index
[junit4:junit4] 2> 372548 T815 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171bfb3e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4e4dc532),segFN=segments_1,generation=1}
[junit4:junit4] 2> 372549 T815 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 372551 T815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 372551 T815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 372552 T815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 372552 T815 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 372553 T815 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 372553 T815 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 372553 T815 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 372554 T815 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 372554 T815 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 372559 T815 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 372563 T815 oass.SolrIndexSearcher.<init> Opening Searcher@74dbf7f3 main
[junit4:junit4] 2> 372563 T815 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/tlog
[junit4:junit4] 2> 372564 T815 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 372564 T815 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 372567 T816 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@74dbf7f3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 372569 T815 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 372569 T815 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 373329 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 373330 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20028_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20028/f"}
[junit4:junit4] 2> 373330 T763 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 373330 T763 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 373344 T814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 373344 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 373344 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 373344 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 373344 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 373572 T815 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 373572 T815 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:20028/f collection:collection1 shard:shard1
[junit4:junit4] 2> 373576 T815 oasc.ZkController.register We are http://127.0.0.1:20028/f/collection1/ and leader is http://127.0.0.1:20022/f/collection1/
[junit4:junit4] 2> 373576 T815 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:20028/f
[junit4:junit4] 2> 373576 T815 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 373577 T815 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C37 name=collection1 org.apache.solr.core.SolrCore@31321838 url=http://127.0.0.1:20028/f/collection1 node=127.0.0.1:20028_f C37_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:20028_f, base_url=http://127.0.0.1:20028/f}
[junit4:junit4] 2> 373577 T817 C37 P20028 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 373578 T817 C37 P20028 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 373578 T815 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 373578 T817 C37 P20028 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 373579 T817 C37 P20028 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 373579 T740 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 373580 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 373580 T817 C37 P20028 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 373581 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 373606 T772 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:20028_f_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 373812 T740 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 373815 T740 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:20032
[junit4:junit4] 2> 373815 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 373816 T740 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 373816 T740 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368656371801
[junit4:junit4] 2> 373817 T740 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368656371801/solr.xml
[junit4:junit4] 2> 373818 T740 oasc.CoreContainer.<init> New CoreContainer 1740448919
[junit4:junit4] 2> 373818 T740 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368656371801/'
[junit4:junit4] 2> 373819 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368656371801/'
[junit4:junit4] 2> 373917 T740 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 373917 T740 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 373918 T740 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 373919 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 373919 T740 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 373920 T740 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 373920 T740 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 373921 T740 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 373921 T740 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 373922 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 373937 T740 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 373938 T740 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:20015/solr
[junit4:junit4] 2> 373939 T740 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 373940 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 373943 T829 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1738ecb8 name:ZooKeeperConnection Watcher:127.0.0.1:20015 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 373943 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 373959 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 373971 T740 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 373973 T831 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@712fe9c1 name:ZooKeeperConnection Watcher:127.0.0.1:20015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 373973 T740 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 373985 T740 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 374849 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 374850 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:20028_f_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20028_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20028/f"}
[junit4:junit4] 2> 374869 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374870 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374869 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374869 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374870 T831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374869 T814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374989 T740 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:20032_f
[junit4:junit4] 2> 374991 T740 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:20032_f
[junit4:junit4] 2> 374994 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374994 T831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374994 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374994 T762 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374994 T814 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374994 T769 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374995 T814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374995 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374996 T831 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374996 T798 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374996 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375007 T782 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 375012 T832 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368656371801/collection1
[junit4:junit4] 2> 375012 T832 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 375013 T832 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 375013 T832 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 375015 T832 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368656371801/collection1/'
[junit4:junit4] 2> 375016 T832 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368656371801/collection1/lib/README' to classloader
[junit4:junit4] 2> 375017 T832 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368656371801/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 375066 T832 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 375123 T832 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 375225 T832 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 375231 T832 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 375609 T772 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:20028_f_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 375609 T772 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:20028_f_collection1&state=recovering&nodeName=127.0.0.1:20028_f&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 375786 T832 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 375791 T832 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 375793 T832 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 375798 T832 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 375828 T832 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 375828 T832 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368656371801/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/
[junit4:junit4] 2> 375829 T832 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f702c48
[junit4:junit4] 2> 375829 T832 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 375830 T832 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4
[junit4:junit4] 2> 375830 T832 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/index/
[junit4:junit4] 2> 375831 T832 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 375831 T832 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/index
[junit4:junit4] 2> 375834 T832 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@58fdb3de lockFactory=org.apache.lucene.store.NativeFSLockFactory@7ad698f7),segFN=segments_1,generation=1}
[junit4:junit4] 2> 375835 T832 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 375838 T832 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 375838 T832 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 375839 T832 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 375840 T832 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 375840 T832 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 375841 T832 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 375841 T832 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 375842 T832 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 375842 T832 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 375850 T832 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 375856 T832 oass.SolrIndexSearcher.<init> Opening Searcher@62bbb074 main
[junit4:junit4] 2> 375856 T832 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/tlog
[junit4:junit4] 2> 375857 T832 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 375858 T832 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 375862 T833 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@62bbb074 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 375865 T832 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 375865 T832 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 376375 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 376376 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20032_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20032/f"}
[junit4:junit4] 2> 376377 T763 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 376377 T763 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 376382 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376382 T831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376382 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376382 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376382 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376382 T814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376894 T832 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 376894 T832 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:20032/f collection:collection1 shard:shard2
[junit4:junit4] 2> 376897 T832 oasc.ZkController.register We are http://127.0.0.1:20032/f/collection1/ and leader is http://127.0.0.1:20025/f/collection1/
[junit4:junit4] 2> 376897 T832 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:20032/f
[junit4:junit4] 2> 376898 T832 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 376898 T832 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C38 name=collection1 org.apache.solr.core.SolrCore@2fa956c0 url=http://127.0.0.1:20032/f/collection1 node=127.0.0.1:20032_f C38_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:20032_f, base_url=http://127.0.0.1:20032/f}
[junit4:junit4] 2> 376899 T834 C38 P20032 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 376899 T832 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 376899 T834 C38 P20032 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 376900 T834 C38 P20032 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 376900 T834 C38 P20032 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 376901 T740 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 376901 T740 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 376902 T834 C38 P20032 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 376902 T740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 376918 T740 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 376919 T788 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:20032_f_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 376919 T740 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 376920 T740 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 376921 T740 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C37_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:20028_f, base_url=http://127.0.0.1:20028/f}
[junit4:junit4] 2> 377611 T817 C37 P20028 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:20022/f/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 377611 T817 C37 P20028 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:20028/f START replicas=[http://127.0.0.1:20022/f/collection1/] nUpdates=100
[junit4:junit4] 2> 377612 T817 C37 P20028 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 377612 T817 C37 P20028 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 377613 T817 C37 P20028 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 377613 T817 C37 P20028 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 377613 T817 C37 P20028 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 377614 T817 C37 P20028 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:20022/f/collection1/. core=collection1
[junit4:junit4] 2> 377614 T817 C37 P20028 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C39 name=collection1 org.apache.solr.core.SolrCore@17fc0cb0 url=http://127.0.0.1:20022/f/collection1 node=127.0.0.1:20022_f C39_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:20022_f, base_url=http://127.0.0.1:20022/f, leader=true}
[junit4:junit4] 2> 377620 T773 C39 P20022 oasc.SolrCore.execute [collection1] webapp=/f path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 377624 T774 C39 P20022 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 377627 T774 C39 P20022 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6c452fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e2da07a),segFN=segments_1,generation=1}
[junit4:junit4] 2> 377628 T774 C39 P20022 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 377628 T774 C39 P20022 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6c452fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e2da07a),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6c452fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e2da07a),segFN=segments_2,generation=2}
[junit4:junit4] 2> 377629 T774 C39 P20022 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 377630 T774 C39 P20022 oass.SolrIndexSearcher.<init> Opening Searcher@3a043413 realtime
[junit4:junit4] 2> 377630 T774 C39 P20022 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 377630 T774 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 377631 T817 C37 P20028 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 377632 T817 C37 P20028 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 377634 T775 C39 P20022 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 377634 T775 C39 P20022 oasc.SolrCore.execute [collection1] webapp=/f path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 377635 T817 C37 P20028 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 377635 T817 C37 P20028 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 377635 T817 C37 P20028 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 377638 T775 C39 P20022 oasc.SolrCore.execute [collection1] webapp=/f path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 377639 T817 C37 P20028 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 377640 T817 C37 P20028 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/index.20130516021935840
[junit4:junit4] 2> 377641 T817 C37 P20028 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@41883161 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b7a3576) fullCopy=false
[junit4:junit4] 2> 377643 T775 C39 P20022 oasc.SolrCore.execute [collection1] webapp=/f path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 377644 T817 C37 P20028 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 377646 T817 C37 P20028 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 377646 T817 C37 P20028 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 377648 T817 C37 P20028 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171bfb3e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4e4dc532),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171bfb3e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4e4dc532),segFN=segments_2,generation=2}
[junit4:junit4] 2> 377648 T817 C37 P20028 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 377649 T817 C37 P20028 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 377649 T817 C37 P20028 oass.SolrIndexSearcher.<init> Opening Searcher@a920de2 main
[junit4:junit4] 2> 377650 T816 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a920de2 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 377651 T817 C37 P20028 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/index.20130516021935840 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/index.20130516021935840;done=true>>]
[junit4:junit4] 2> 377651 T817 C37 P20028 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/index.20130516021935840
[junit4:junit4] 2> 377652 T817 C37 P20028 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty3/index.20130516021935840
[junit4:junit4] 2> 377652 T817 C37 P20028 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 377652 T817 C37 P20028 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 377653 T817 C37 P20028 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 377653 T817 C37 P20028 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 377658 T817 C37 P20028 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 377887 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 377888 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:20032_f_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20032_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20032/f"}
[junit4:junit4] 2> 377892 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:20028_f_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20028_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20028/f"}
[junit4:junit4] 2> 377906 T814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 377906 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 377906 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 377907 T831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 377906 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 377906 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 377920 T788 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:20032_f_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 377920 T788 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:20032_f_collection1&state=recovering&nodeName=127.0.0.1:20032_f&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 377923 T740 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 378925 T740 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C38_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:20032_f, base_url=http://127.0.0.1:20032/f}
[junit4:junit4] 2> 379922 T834 C38 P20032 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:20025/f/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 379922 T834 C38 P20032 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:20032/f START replicas=[http://127.0.0.1:20025/f/collection1/] nUpdates=100
[junit4:junit4] 2> 379923 T834 C38 P20032 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 379923 T834 C38 P20032 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 379923 T834 C38 P20032 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 379924 T834 C38 P20032 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 379925 T834 C38 P20032 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 379925 T834 C38 P20032 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:20025/f/collection1/. core=collection1
[junit4:junit4] 2> 379925 T834 C38 P20032 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 379928 T740 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C40 name=collection1 org.apache.solr.core.SolrCore@5935b538 url=http://127.0.0.1:20025/f/collection1 node=127.0.0.1:20025_f C40_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:20025_f, base_url=http://127.0.0.1:20025/f, leader=true}
[junit4:junit4] 2> 379932 T789 C40 P20025 oasc.SolrCore.execute [collection1] webapp=/f path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 379935 T790 C40 P20025 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 379938 T790 C40 P20025 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e205b65 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f3ea022),segFN=segments_1,generation=1}
[junit4:junit4] 2> 379938 T790 C40 P20025 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 379939 T790 C40 P20025 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e205b65 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f3ea022),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e205b65 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f3ea022),segFN=segments_2,generation=2}
[junit4:junit4] 2> 379940 T790 C40 P20025 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 379940 T790 C40 P20025 oass.SolrIndexSearcher.<init> Opening Searcher@16b50694 realtime
[junit4:junit4] 2> 379941 T790 C40 P20025 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 379941 T790 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 379942 T834 C38 P20032 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 379942 T834 C38 P20032 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 379944 T791 C40 P20025 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 379944 T791 C40 P20025 oasc.SolrCore.execute [collection1] webapp=/f path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 379945 T834 C38 P20032 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 379945 T834 C38 P20032 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 379945 T834 C38 P20032 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 379948 T791 C40 P20025 oasc.SolrCore.execute [collection1] webapp=/f path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 379949 T834 C38 P20032 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 379950 T834 C38 P20032 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/index.20130516021938149
[junit4:junit4] 2> 379950 T834 C38 P20032 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@4177fc5c lockFactory=org.apache.lucene.store.NativeFSLockFactory@337ea934) fullCopy=false
[junit4:junit4] 2> 379953 T791 C40 P20025 oasc.SolrCore.execute [collection1] webapp=/f path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 379954 T834 C38 P20032 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 379955 T834 C38 P20032 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 379955 T834 C38 P20032 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 379957 T834 C38 P20032 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@58fdb3de lockFactory=org.apache.lucene.store.NativeFSLockFactory@7ad698f7),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@58fdb3de lockFactory=org.apache.lucene.store.NativeFSLockFactory@7ad698f7),segFN=segments_2,generation=2}
[junit4:junit4] 2> 379958 T834 C38 P20032 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 379958 T834 C38 P20032 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 379958 T834 C38 P20032 oass.SolrIndexSearcher.<init> Opening Searcher@3b1c239c main
[junit4:junit4] 2> 379960 T833 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b1c239c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 379960 T834 C38 P20032 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/index.20130516021938149 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/index.20130516021938149;done=true>>]
[junit4:junit4] 2> 379961 T834 C38 P20032 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/index.20130516021938149
[junit4:junit4] 2> 379961 T834 C38 P20032 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368656355583/jetty4/index.20130516021938149
[junit4:junit4] 2> 379961 T834 C38 P20032 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 379961 T834 C38 P20032 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 379962 T834 C38 P20032 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 379962 T834 C38 P20032 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 379994 T834 C38 P20032 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 380922 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 380923 T763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:20032_f_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20032_f",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20032/f"}
[junit4:junit4] 2> 380936 T798 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 380936 T831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 380937 T740 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 380936 T782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 380936 T762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 380936 T814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 380936 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 380939 T740 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C41 name=collection1 org.apache.solr.core.SolrCore@69c7f04d url=http://127.0.0.1:20018/f/collection1 node=127.0.0.1:20018_f C41_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:20018_f, base_url=http://127.0.0.1:20018/f, leader=true}
[junit4:junit4] 2> 380949 T752 C41 P20018 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 380953 T752 C41 P20018 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2c786e17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1017e84b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 380953 T752 C41 P20018 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 380954 T752 C41 P20018 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2c786e17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1017e84b),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2c786e17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1017e84b),segFN=segments_2,generation=2}
[junit4:junit4] 2> 380955 T752 C41 P20018 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 380955 T752 C41 P20018 oass.SolrIndexSearcher.<init> Opening Searcher@39290a1d main
[junit4:junit4] 2> 380956 T752 C41 P20018 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 380957 T766 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39290a1d main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 380957 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 380970 T792 C40 P20025 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 380972 T792 C40 P20025 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e205b65 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f3ea022),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e205b65 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f3ea022),segFN=segments_3,generation=3}
[junit4:junit4] 2> 380972 T792 C40 P20025 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 380973 T792 C40 P20025 oass.SolrIndexSearcher.<init> Opening Searcher@7cf42466 main
[junit4:junit4] 2> 380973 T792 C40 P20025 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 380974 T800 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7cf42466 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 380975 T792 C40 P20025 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:20022/f/collection1/, StdNode: http://127.0.0.1:20028/f/collection1/, StdNode: http://127.0.0.1:20032/f/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 380978 T776 C39 P20022 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 380980 T776 C39 P20022 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6c452fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e2da07a),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6c452fb4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2e2da07a),segFN=segments_3,generation=3}
[junit4:junit4] 2> 380981 T776 C39 P20022 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 380982 T776 C39 P20022 oass.SolrIndexSearcher.<init> Opening Searcher@5135c387 main
[junit4:junit4] 2> 380983 T776 C39 P20022 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 380983 T784 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5135c387 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 380984 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> C38_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:20032_f, base_url=http://127.0.0.1:20032/f}
[junit4:junit4] 2> 380986 T821 C38 P20032 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C37_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:20028_f, base_url=http://127.0.0.1:20028/f}
[junit4:junit4] 2> 380986 T804 C37 P20028 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 380987 T821 C38 P20032 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@58fdb3de lockFactory=org.apache.lucene.store.NativeFSLockFactory@7ad698f7),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@58fdb3de lockFactory=org.apache.lucene.store.NativeFSLockFactory@7ad698f7),segFN=segments_3,generation=3}
[junit4:junit4] 2> 380987 T804 C37 P20028 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171bfb3e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4e4dc532),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171bfb3e lockFactory=org.apache.lucene.store.NativeFSLockFactory@4e4dc532),segFN=segments_3,generation=3}
[junit4:junit4] 2> 380988 T821 C38 P20032 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 380988 T804 C37 P20028 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 380989 T821 C38 P20032 oass.SolrIndexSearcher.<init> Opening Searcher@4b7dd928 main
[junit4:junit4] 2> 380989 T804 C37 P20028 oass.SolrIndexSearcher.<init> Opening Searcher@260a92f7 main
[junit4:junit4] 2> 380990 T821 C38 P20032 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 380990 T833 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b7dd928 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 380990 T804 C37 P20028 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 380991 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 380991 T816 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@260a92f7 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 380992 T804 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 380993 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 23
[junit4:junit4] 2> 380994 T740 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 380997 T777 C39 P20022 oasc.SolrCore.execute [collection1] webapp=/f path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 380999 T805 C37 P20028 oasc.SolrCore.execute [collection1] webapp=/f path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 381001 T793 C40 P20025 oasc.SolrCore.execute [collection1] webapp=/f path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 381003 T822 C38 P20032 oasc.SolrCore.execute [collection1] webapp=/f path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 383008 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435140233580511232)} 0 2
[junit4:junit4] 2> 383017 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={update.distrib=FROMLEADER&_version_=-1435140233584705536&update.from=http://127.0.0.1:20025/f/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435140233584705536)} 0 2
[junit4:junit4] 2> 383020 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={update.distrib=FROMLEADER&_version_=-1435140233588899840&update.from=http://127.0.0.1:20022/f/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435140233588899840)} 0 1
[junit4:junit4] 2> 383021 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1435140233588899840)} 0 6
[junit4:junit4] 2> 383022 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435140233584705536)} 0 11
[junit4:junit4] 2> 383026 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[0 (1435140233599385600)]} 0 1
[junit4:junit4] 2> 383032 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1435140233602531328)]} 0 1
[junit4:junit4] 2> 383032 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[0 (1435140233602531328)]} 0 4
[junit4:junit4] 2> 383037 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[1 (1435140233610919936)]} 0 1
[junit4:junit4] 2> 383044 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1435140233615114240)]} 0 1
[junit4:junit4] 2> 383044 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[1 (1435140233615114240)]} 0 4
[junit4:junit4] 2> 383048 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[2 (1435140233622454272)]} 0 1
[junit4:junit4] 2> 383055 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1435140233626648576)]} 0 1
[junit4:junit4] 2> 383055 T794 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[2 (1435140233626648576)]} 0 4
[junit4:junit4] 2> 383056 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[2]} 0 6
[junit4:junit4] 2> 383059 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[3 (1435140233635037184)]} 0 0
[junit4:junit4] 2> 383065 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1435140233638182912)]} 0 0
[junit4:junit4] 2> 383066 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[3 (1435140233638182912)]} 0 4
[junit4:junit4] 2> 383069 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[4 (1435140233645522944)]} 0 0
[junit4:junit4] 2> 383077 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1435140233650765824)]} 0 0
[junit4:junit4] 2> 383078 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1435140233650765824)]} 0 4
[junit4:junit4] 2> 383079 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[4]} 0 7
[junit4:junit4] 2> 383081 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[5 (1435140233658105856)]} 0 0
[junit4:junit4] 2> 383088 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1435140233661251584)]} 0 1
[junit4:junit4] 2> 383088 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[5 (1435140233661251584)]} 0 4
[junit4:junit4] 2> 383091 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[6 (1435140233668591616)]} 0 0
[junit4:junit4] 2> 383097 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1435140233671737344)]} 0 0
[junit4:junit4] 2> 383098 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[6 (1435140233671737344)]} 0 4
[junit4:junit4] 2> 383101 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[7 (1435140233679077376)]} 0 0
[junit4:junit4] 2> 383108 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1435140233684320256)]} 0 0
[junit4:junit4] 2> 383109 T794 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[7 (1435140233684320256)]} 0 3
[junit4:junit4] 2> 383110 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[7]} 0 6
[junit4:junit4] 2> 383112 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[8 (1435140233690611712)]} 0 0
[junit4:junit4] 2> 383119 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1435140233695854592)]} 0 0
[junit4:junit4] 2> 383120 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[8 (1435140233695854592)]} 0 3
[junit4:junit4] 2> 383121 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[8]} 0 6
[junit4:junit4] 2> 383123 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[9 (1435140233702146048)]} 0 0
[junit4:junit4] 2> 383131 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1435140233707388928)]} 0 0
[junit4:junit4] 2> 383132 T794 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[9 (1435140233707388928)]} 0 4
[junit4:junit4] 2> 383133 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[9]} 0 7
[junit4:junit4] 2> 383135 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[10 (1435140233714728960)]} 0 0
[junit4:junit4] 2> 383142 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1435140233719971840)]} 0 0
[junit4:junit4] 2> 383143 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[10 (1435140233719971840)]} 0 3
[junit4:junit4] 2> 383144 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[10]} 0 6
[junit4:junit4] 2> 383146 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[11 (1435140233726263296)]} 0 0
[junit4:junit4] 2> 383153 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1435140233731506176)]} 0 0
[junit4:junit4] 2> 383154 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[11 (1435140233731506176)]} 0 3
[junit4:junit4] 2> 383155 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[11]} 0 6
[junit4:junit4] 2> 383157 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[12 (1435140233737797632)]} 0 0
[junit4:junit4] 2> 383164 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1435140233743040512)]} 0 0
[junit4:junit4] 2> 383165 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[12 (1435140233743040512)]} 0 3
[junit4:junit4] 2> 383166 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[12]} 0 6
[junit4:junit4] 2> 383168 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[13 (1435140233749331968)]} 0 0
[junit4:junit4] 2> 383176 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1435140233754574848)]} 0 1
[junit4:junit4] 2> 383176 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[13 (1435140233754574848)]} 0 3
[junit4:junit4] 2> 383177 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[13]} 0 6
[junit4:junit4] 2> 383180 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[14 (1435140233761914880)]} 0 0
[junit4:junit4] 2> 383186 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1435140233765060608)]} 0 0
[junit4:junit4] 2> 383187 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[14 (1435140233765060608)]} 0 4
[junit4:junit4] 2> 383190 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[15 (1435140233772400640)]} 0 0
[junit4:junit4] 2> 383195 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1435140233775546368)]} 0 0
[junit4:junit4] 2> 383196 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[15 (1435140233775546368)]} 0 3
[junit4:junit4] 2> 383199 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[16 (1435140233781837824)]} 0 0
[junit4:junit4] 2> 383206 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1435140233787080704)]} 0 0
[junit4:junit4] 2> 383207 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[16 (1435140233787080704)]} 0 3
[junit4:junit4] 2> 383208 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[16]} 0 6
[junit4:junit4] 2> 383210 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[17 (1435140233793372160)]} 0 0
[junit4:junit4] 2> 383216 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1435140233796517888)]} 0 0
[junit4:junit4] 2> 383217 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[17 (1435140233796517888)]} 0 4
[junit4:junit4] 2> 383220 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[18 (1435140233803857920)]} 0 0
[junit4:junit4] 2> 383228 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1435140233809100800)]} 0 0
[junit4:junit4] 2> 383229 T794 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[18 (1435140233809100800)]} 0 4
[junit4:junit4] 2> 383230 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[18]} 0 7
[junit4:junit4] 2> 383233 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[19 (1435140233817489408)]} 0 0
[junit4:junit4] 2> 383239 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1435140233820635136)]} 0 0
[junit4:junit4] 2> 383240 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[19 (1435140233820635136)]} 0 4
[junit4:junit4] 2> 383243 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[20 (1435140233827975168)]} 0 0
[junit4:junit4] 2> 383251 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1435140233833218048)]} 0 0
[junit4:junit4] 2> 383252 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[20 (1435140233833218048)]} 0 4
[junit4:junit4] 2> 383253 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[20]} 0 7
[junit4:junit4] 2> 383256 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[21 (1435140233841606656)]} 0 0
[junit4:junit4] 2> 383261 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1435140233844752384)]} 0 0
[junit4:junit4] 2> 383262 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[21 (1435140233844752384)]} 0 3
[junit4:junit4] 2> 383265 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[22 (1435140233851043840)]} 0 0
[junit4:junit4] 2> 383270 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1435140233854189568)]} 0 0
[junit4:junit4] 2> 383271 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[22 (1435140233854189568)]} 0 3
[junit4:junit4] 2> 383274 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[23 (1435140233860481024)]} 0 0
[junit4:junit4] 2> 383282 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1435140233865723904)]} 0 0
[junit4:junit4] 2> 383283 T794 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[23 (1435140233865723904)]} 0 4
[junit4:junit4] 2> 383284 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[23]} 0 7
[junit4:junit4] 2> 383287 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[24 (1435140233874112512)]} 0 0
[junit4:junit4] 2> 383295 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1435140233879355392)]} 0 0
[junit4:junit4] 2> 383296 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[24 (1435140233879355392)]} 0 4
[junit4:junit4] 2> 383297 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[24]} 0 7
[junit4:junit4] 2> 383300 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[25 (1435140233887744000)]} 0 0
[junit4:junit4] 2> 383305 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1435140233890889728)]} 0 0
[junit4:junit4] 2> 383306 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[25 (1435140233890889728)]} 0 3
[junit4:junit4] 2> 383309 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[26 (1435140233897181184)]} 0 0
[junit4:junit4] 2> 383317 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1435140233902424064)]} 0 0
[junit4:junit4] 2> 383318 T794 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[26 (1435140233902424064)]} 0 4
[junit4:junit4] 2> 383319 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[26]} 0 7
[junit4:junit4] 2> 383321 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[27 (1435140233909764096)]} 0 0
[junit4:junit4] 2> 383326 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1435140233912909824)]} 0 0
[junit4:junit4] 2> 383327 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[27 (1435140233912909824)]} 0 3
[junit4:junit4] 2> 383330 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[28 (1435140233919201280)]} 0 0
[junit4:junit4] 2> 383335 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1435140233922347008)]} 0 0
[junit4:junit4] 2> 383336 T778 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[28 (1435140233922347008)]} 0 3
[junit4:junit4] 2> 383339 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[29 (1435140233928638464)]} 0 0
[junit4:junit4] 2> 383345 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1435140233931784192)]} 0 0
[junit4:junit4] 2> 383346 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[29 (1435140233931784192)]} 0 4
[junit4:junit4] 2> 383349 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[30 (1435140233939124224)]} 0 0
[junit4:junit4] 2> 383354 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1435140233942269952)]} 0 0
[junit4:junit4] 2> 383355 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[30 (1435140233942269952)]} 0 3
[junit4:junit4] 2> 383358 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[31 (1435140233948561408)]} 0 0
[junit4:junit4] 2> 383363 T821 C38 P20032 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1435140233951707136)]} 0 0
[junit4:junit4] 2> 383364 T792 C40 P20025 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[31 (1435140233951707136)]} 0 3
[junit4:junit4] 2> 383367 T752 C41 P20018 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={wt=javabin&version=2} {add=[32 (1435140233957998592)]} 0 0
[junit4:junit4] 2> 383375 T806 C37 P20028 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20022/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1435140233963241472)]} 0 0
[junit4:junit4] 2> 383376 T776 C39 P20022 oasup.LogUpdateProcessor.finish [collection1] webapp=/f path=/update params={distrib.from=http://127.0.0.1:20025/f/collection1/&upd
[...truncated too long message...]
[junit4:junit4] 2> 1933190 T2368 C140 P57375 oasu.TransactionLog.forceClose ERROR Error: Forcing close of tlog{file=./org.apache.solr.cloud.SyncSliceTest-1368657019983/jetty4/tlog/tlog.0000000000000000005 refcount=2}
[junit4:junit4] 2> 1933190 T2368 C140 P57375 oasu.TransactionLog.forceClose ERROR Error: Forcing close of tlog{file=./org.apache.solr.cloud.SyncSliceTest-1368657019983/jetty4/tlog/tlog.0000000000000000001 refcount=2}
[junit4:junit4] 2> 1933193 T2368 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1933194 T2368 oasc.CachingDirectoryFactory.close Closing NRTCachingDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1933194 T2368 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1368657019983/jetty4/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1368657019983/jetty4/index;done=false>>]
[junit4:junit4] 2> 1933195 T2368 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1368657019983/jetty4/index
[junit4:junit4] 2> 1933195 T2368 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1368657019983/jetty4 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1368657019983/jetty4;done=false>>]
[junit4:junit4] 2> 1933195 T2368 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1368657019983/jetty4
[junit4:junit4] 2> 1933195 T2368 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 1933196 T2368 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election/89696306461999116-127.0.0.1:57375_g%2Fd_collection1-n_0000000006
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:152)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:63)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:246)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1933196 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1933197 T2368 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1933197 T2368 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1933197 T2368 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Direct), text=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=Direct), rnd_b=Pulsing41(freqCutoff=16 minBlockSize=95 maxBlockSize=210), intDefault=PostingsFormat(name=Direct), id=Pulsing41(freqCutoff=16 minBlockSize=95 maxBlockSize=210), timestamp=PostingsFormat(name=Direct), a_t=PostingsFormat(name=Direct), range_facet_sl=PostingsFormat(name=Memory doPackFST= true), range_facet_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), other_tl1=PostingsFormat(name=Direct), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings)}, docValues:{}, sim=DefaultSimilarity, locale=el_GR, timezone=Europe/Vienna
[junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=4,free=146086984,total=420937728
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestDFRSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestCodecSupport, PingRequestHandlerTest, URLClassifyProcessorTest, DefaultValueUpdateProcessorTest, TestWordDelimiterFilterFactory, FullSolrCloudDistribCmdsTest, ChaosMonkeySafeLeaderTest, TestDistributedSearch, ClusterStateTest, FieldAnalysisRequestHandlerTest, EchoParamsTest, BadIndexSchemaTest, TestElisionMultitermQuery, PreAnalyzedUpdateProcessorTest, DebugComponentTest, HighlighterConfigTest, DistanceFunctionTest, FileUtilsTest, TestCopyFieldCollectionResource, ChaosMonkeyShardSplitTest, SimplePostToolTest, SampleTest, TestSystemIdResolver, TestLuceneMatchVersion, HardAutoCommitTest, DirectSolrSpellCheckerTest, StandardRequestHandlerTest, ConvertedLegacyTest, TestBinaryResponseWriter, TestHashPartitioner, CollectionsAPIDistributedZkTest, TestQueryTypes, XmlUpdateRequestHandlerTest, SolrCmdDistributorTest, TestSolr4Spatial, TestUtils, LeaderElectionIntegrationTest, TestShardHandlerFactory, SolrRequestParserTest, TestArbitraryIndexDir, SpellCheckCollatorTest, TestSchemaNameResource, SolrPluginUtilsTest, TestCSVLoader, TestDefaultSearchFieldResource, UpdateParamsTest, TestZkChroot, RequiredFieldsTest, SignatureUpdateProcessorFactoryTest, TestRandomFaceting, TestNumberUtils, TestBinaryField, TestDistributedGrouping, TestSolrQueryParserResource, SolrIndexConfigTest, TestComponentsName, TestDynamicFieldCollectionResource, TestJmxIntegration, SpellCheckComponentTest, MultiTermTest, TestUniqueKeyFieldResource, TestOmitPositions, SpellPossibilityIteratorTest, TestCSVResponseWriter, TestSolrJ, CurrencyFieldOpenExchangeTest, TestMergePolicyConfig, SortByFunctionTest, OpenExchangeRatesOrgProviderTest, StatsComponentTest, UnloadDistributedZkTest, SyncSliceTest]
[junit4:junit4] Completed on J1 in 911.77s, 1 test, 1 failure <<< FAILURES!
[...truncated 60 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:383: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:363: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1243: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:887: There were test failures: 296 suites, 1234 tests, 1 suite-level error, 1 error, 1 failure, 27 ignored (8 assumptions)
Total time: 54 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still Failing
Posted by Adrien Grand <jp...@gmail.com>.
Since the resolution of this failure doesn't look trivial (as there
are several options), I opened
https://issues.apache.org/jira/browse/LUCENE-5004.
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still Failing
Posted by Shai Erera <se...@gmail.com>.
Looking at the code, I think that today flush() cannot throw diskFullEx?
MockIO checks for diskFull in writeBytes and copyBytes, using
dir.sizeInBytes.
So I think if you limit maxSize=5 and write 4 + 4 bytes, to a Directory
with a buffer of 20 bytes, then both writeBytes will succeed (as
dir.sizeInBytes = 0).
flush() will also succeed since it only calls delegate.flush().
So I think either MDW should track its size, or MockIO track the num
written bytes?
Shai
On Thu, May 16, 2013 at 7:08 PM, Robert Muir <rc...@gmail.com> wrote:
> On Thu, May 16, 2013 at 8:45 AM, Adrien Grand <jp...@gmail.com> wrote:
>
> > Using the Mock IndexOutput to track bytes is an option, but I was
> > thinking it could be interesting too to see what happens with
> > directories that buffer content so that the disk full exception
> > happens in flush instead of writeBytes?
> >
>
> But it can easily happen in both places...
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still Failing
Posted by Robert Muir <rc...@gmail.com>.
On Thu, May 16, 2013 at 8:45 AM, Adrien Grand <jp...@gmail.com> wrote:
> Using the Mock IndexOutput to track bytes is an option, but I was
> thinking it could be interesting too to see what happens with
> directories that buffer content so that the disk full exception
> happens in flush instead of writeBytes?
>
But it can easily happen in both places...
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still Failing
Posted by Shai Erera <se...@gmail.com>.
I think this exception should be thrown only when the bytes land in the
Directory. I think that in general we buffer bytes before sending them to
the actual IndexOutput? I don't know if this is done with RAMDir too i.e.
that there isn't code out there that buffers its writes to any IndexOutput,
not caring if it's RAMFile or not.
And technically, if you have a buffer of 16K, but can only write 2K to the
underlying Directory, you shouldn't hit the exception until you actually
flush the bytes?
Adrien, not in front of the code now, but if writeBytes applies the limit,
why do we need any special logic in MDW.flush?
Shai
On Thu, May 16, 2013 at 3:45 PM, Adrien Grand <jp...@gmail.com> wrote:
> On Thu, May 16, 2013 at 1:58 PM, Robert Muir <rc...@gmail.com> wrote:
> > I dont get it. MDW wraps its IndexOutput so it "knows"... sounds like
> > the counting is off.
>
> The problem is that RAMDirectory delays the counting.
> MockDirectoryWrapper.getRecomputedActualSizeInBytes sums all the
> lengths of the existing RAMFiles to get the actual size, but
> RAMFile.length is only updated after a RAMOutputStream seek or flush.
> This means that if you write 5 bytes, then 3 bytes, RAMFile.length
> will still be 0 and then suddenly upon flush it will become 5+3=8.
>
> Using the Mock IndexOutput to track bytes is an option, but I was
> thinking it could be interesting too to see what happens with
> directories that buffer content so that the disk full exception
> happens in flush instead of writeBytes?
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still Failing
Posted by Adrien Grand <jp...@gmail.com>.
On Thu, May 16, 2013 at 1:58 PM, Robert Muir <rc...@gmail.com> wrote:
> I dont get it. MDW wraps its IndexOutput so it "knows"... sounds like
> the counting is off.
The problem is that RAMDirectory delays the counting.
MockDirectoryWrapper.getRecomputedActualSizeInBytes sums all the
lengths of the existing RAMFiles to get the actual size, but
RAMFile.length is only updated after a RAMOutputStream seek or flush.
This means that if you write 5 bytes, then 3 bytes, RAMFile.length
will still be 0 and then suddenly upon flush it will become 5+3=8.
Using the Mock IndexOutput to track bytes is an option, but I was
thinking it could be interesting too to see what happens with
directories that buffer content so that the disk full exception
happens in flush instead of writeBytes?
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still Failing
Posted by Robert Muir <rc...@gmail.com>.
I dont get it. MDW wraps its IndexOutput so it "knows"... sounds like
the counting is off.
On Thu, May 16, 2013 at 6:24 AM, Adrien Grand <jp...@gmail.com> wrote:
> On Thu, May 16, 2013 at 11:30 AM, Shai Erera <se...@gmail.com> wrote:
>> As long as bytes are buffered, I think it's OK to not hit disk-full... they
>> never made it to the directory yet.
>
> Good point. So the test needs to call commit right after
> IndexWriter.add(Document) to make sure flush gets called, and
> MockDirectoryWrapper.flush needs to check disk full right after
> delegate.flush?
>
> --
> Adrien
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still Failing
Posted by Adrien Grand <jp...@gmail.com>.
On Thu, May 16, 2013 at 11:30 AM, Shai Erera <se...@gmail.com> wrote:
> As long as bytes are buffered, I think it's OK to not hit disk-full... they
> never made it to the directory yet.
Good point. So the test needs to call commit right after
IndexWriter.add(Document) to make sure flush gets called, and
MockDirectoryWrapper.flush needs to check disk full right after
delegate.flush?
--
Adrien
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still Failing
Posted by Shai Erera <se...@gmail.com>.
I hit a similar failure with the Replicator tests, or should I say "did not
hit the expected disk full".
I added a test to TestMDW.testDiskFull (fixing copyBytes to fail on
disk-full) but I put a comment on why you need to call flush().
Basically, you should call flush to ensure that bytes are not buffered.
Especially in a test which verifies 'disk full'.
As long as bytes are buffered, I think it's OK to not hit disk-full... they
never made it to the directory yet.
Shai
On Thu, May 16, 2013 at 12:12 PM, Adrien Grand <jp...@gmail.com> wrote:
> I can reproduce this one. It seems to me that the problem is that
> MockDirectoryWrapper.getRecomputedSizeInBytes uses RAMFile.length
> although RAMFile.length is only set on flush or seek and is 0 until
> then? Should setFileLength be called after every write?
>
> On Thu, May 16, 2013 at 8:35 AM, Apache Jenkins Server
> <je...@builds.apache.org> wrote:
> > Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/1623/
> >
> > 1 tests failed.
> > REGRESSION:
> org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull
> >
> > Error Message:
> > did not hit disk full
> >
> > Stack Trace:
> > java.lang.AssertionError: did not hit disk full
> > at
> __randomizedtesting.SeedInfo.seed([9159284DB5A0A12D:1FBA21C9A14FE9]:0)
> > at org.junit.Assert.fail(Assert.java:93)
> > at
> org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull(TestIndexWriterOnDiskFull.java:537)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:616)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> > at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> > at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> > at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> > at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> > at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> > at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> > at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> > at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> > at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> > at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> > at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> > at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> > at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> > at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> > at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> > at java.lang.Thread.run(Thread.java:679)
> >
> >
> >
> >
> > Build Log:
> > [...truncated 205 lines...]
> > [junit4:junit4] Suite: org.apache.lucene.index.TestIndexWriterOnDiskFull
> > [junit4:junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexWriterOnDiskFull -Dtests.method=testImmediateDiskFull
> -Dtests.seed=9159284DB5A0A12D -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ar -Dtests.timezone=America/Managua
> -Dtests.file.encoding=ISO8859-1
> > [junit4:junit4] FAILURE 0.51s J0 |
> TestIndexWriterOnDiskFull.testImmediateDiskFull <<<
> > [junit4:junit4] > Throwable #1: java.lang.AssertionError: did not hit
> disk full
> > [junit4:junit4] > at
> __randomizedtesting.SeedInfo.seed([9159284DB5A0A12D:1FBA21C9A14FE9]:0)
> > [junit4:junit4] > at
> org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull(TestIndexWriterOnDiskFull.java:537)
> > [junit4:junit4] > at java.lang.Thread.run(Thread.java:679)
> > [junit4:junit4] 2> NOTE: test params are: codec=SimpleText,
> sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {f=IB
> SPL-D3(800.0), id=DFR I(F)L1, content=DFR I(ne)2, field=DFR I(n)B1},
> locale=ar, timezone=America/Managua
> > [junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems
> Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=401453760,total=514064384
> > [junit4:junit4] 2> NOTE: All tests run in this JVM:
> [TestIndexWriterOnDiskFull]
> > [junit4:junit4] Completed on J0 in 3.81s, 4 tests, 1 failure <<<
> FAILURES!
> >
> > [...truncated 1161 lines...]
> > BUILD FAILED
> >
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:383:
> The following error occurred while executing this line:
> >
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:363:
> The following error occurred while executing this line:
> >
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39:
> The following error occurred while executing this line:
> >
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build.xml:49:
> The following error occurred while executing this line:
> >
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1243:
> The following error occurred while executing this line:
> >
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:887:
> There were test failures: 364 suites, 2307 tests, 1 failure, 46 ignored (34
> assumptions)
> >
> > Total time: 8 minutes 32 seconds
> > Build step 'Invoke Ant' marked build as failure
> > Archiving artifacts
> > Recording test results
> > Email was triggered for: Failure
> > Sending email for trigger: Failure
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still Failing
Posted by Adrien Grand <jp...@gmail.com>.
I can reproduce this one. It seems to me that the problem is that
MockDirectoryWrapper.getRecomputedSizeInBytes uses RAMFile.length
although RAMFile.length is only set on flush or seek and is 0 until
then? Should setFileLength be called after every write?
On Thu, May 16, 2013 at 8:35 AM, Apache Jenkins Server
<je...@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/1623/
>
> 1 tests failed.
> REGRESSION: org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull
>
> Error Message:
> did not hit disk full
>
> Stack Trace:
> java.lang.AssertionError: did not hit disk full
> at __randomizedtesting.SeedInfo.seed([9159284DB5A0A12D:1FBA21C9A14FE9]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull(TestIndexWriterOnDiskFull.java:537)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at java.lang.Thread.run(Thread.java:679)
>
>
>
>
> Build Log:
> [...truncated 205 lines...]
> [junit4:junit4] Suite: org.apache.lucene.index.TestIndexWriterOnDiskFull
> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterOnDiskFull -Dtests.method=testImmediateDiskFull -Dtests.seed=9159284DB5A0A12D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar -Dtests.timezone=America/Managua -Dtests.file.encoding=ISO8859-1
> [junit4:junit4] FAILURE 0.51s J0 | TestIndexWriterOnDiskFull.testImmediateDiskFull <<<
> [junit4:junit4] > Throwable #1: java.lang.AssertionError: did not hit disk full
> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([9159284DB5A0A12D:1FBA21C9A14FE9]:0)
> [junit4:junit4] > at org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull(TestIndexWriterOnDiskFull.java:537)
> [junit4:junit4] > at java.lang.Thread.run(Thread.java:679)
> [junit4:junit4] 2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {f=IB SPL-D3(800.0), id=DFR I(F)L1, content=DFR I(ne)2, field=DFR I(n)B1}, locale=ar, timezone=America/Managua
> [junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=401453760,total=514064384
> [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestIndexWriterOnDiskFull]
> [junit4:junit4] Completed on J0 in 3.81s, 4 tests, 1 failure <<< FAILURES!
>
> [...truncated 1161 lines...]
> BUILD FAILED
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:383: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:363: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build.xml:49: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1243: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:887: There were test failures: 364 suites, 2307 tests, 1 failure, 46 ignored (34 assumptions)
>
> Total time: 8 minutes 32 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1623 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/1623/
1 tests failed.
REGRESSION: org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull
Error Message:
did not hit disk full
Stack Trace:
java.lang.AssertionError: did not hit disk full
at __randomizedtesting.SeedInfo.seed([9159284DB5A0A12D:1FBA21C9A14FE9]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull(TestIndexWriterOnDiskFull.java:537)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
Build Log:
[...truncated 205 lines...]
[junit4:junit4] Suite: org.apache.lucene.index.TestIndexWriterOnDiskFull
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterOnDiskFull -Dtests.method=testImmediateDiskFull -Dtests.seed=9159284DB5A0A12D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar -Dtests.timezone=America/Managua -Dtests.file.encoding=ISO8859-1
[junit4:junit4] FAILURE 0.51s J0 | TestIndexWriterOnDiskFull.testImmediateDiskFull <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: did not hit disk full
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([9159284DB5A0A12D:1FBA21C9A14FE9]:0)
[junit4:junit4] > at org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull(TestIndexWriterOnDiskFull.java:537)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:679)
[junit4:junit4] 2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {f=IB SPL-D3(800.0), id=DFR I(F)L1, content=DFR I(ne)2, field=DFR I(n)B1}, locale=ar, timezone=America/Managua
[junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=401453760,total=514064384
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestIndexWriterOnDiskFull]
[junit4:junit4] Completed on J0 in 3.81s, 4 tests, 1 failure <<< FAILURES!
[...truncated 1161 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:383: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:363: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build.xml:49: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1243: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:887: There were test failures: 364 suites, 2307 tests, 1 failure, 46 ignored (34 assumptions)
Total time: 8 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1622 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/1622/
2 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:15398 returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:15398 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([4CA14552DE06E6C0:CD47CB4AA95986FC]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
FAILED: junit.framework.TestSuite.org.apache.solr.update.processor.RegexBoostProcessorTest
Error Message:
Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 22,137,840 bytes (threshold is 10,485,760). Field reference sizes (counted individually): - 22,751,288 bytes, protected static org.apache.solr.servlet.SolrRequestParsers org.apache.solr.update.processor.RegexBoostProcessorTest._parser - 5,256 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessorFactory org.apache.solr.update.processor.RegexBoostProcessorTest.factory - 4,568 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessor org.apache.solr.update.processor.RegexBoostProcessorTest.reProcessor - 856 bytes, protected static org.apache.solr.common.params.ModifiableSolrParams org.apache.solr.update.processor.RegexBoostProcessorTest.parameters - 296 bytes, protected static java.lang.String org.apache.solr.SolrTestCaseJ4.testSolrHome - 272 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules - 136 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp - 72 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName
Stack Trace:
junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 22,137,840 bytes (threshold is 10,485,760). Field reference sizes (counted individually):
- 22,751,288 bytes, protected static org.apache.solr.servlet.SolrRequestParsers org.apache.solr.update.processor.RegexBoostProcessorTest._parser
- 5,256 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessorFactory org.apache.solr.update.processor.RegexBoostProcessorTest.factory
- 4,568 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessor org.apache.solr.update.processor.RegexBoostProcessorTest.reProcessor
- 856 bytes, protected static org.apache.solr.common.params.ModifiableSolrParams org.apache.solr.update.processor.RegexBoostProcessorTest.parameters
- 296 bytes, protected static java.lang.String org.apache.solr.SolrTestCaseJ4.testSolrHome
- 272 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules
- 136 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp
- 72 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName
at __randomizedtesting.SeedInfo.seed([4CA14552DE06E6C0]:0)
at com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.java:127)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
Build Log:
[...truncated 8719 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4] 2> 271298 T695 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 271302 T695 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyShardSplitTest-1368666933250
[junit4:junit4] 2> 271303 T695 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 271304 T696 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 271404 T695 oasc.ZkTestServer.run start zk server on port:27978
[junit4:junit4] 2> 271405 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 271414 T702 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a67acf3 name:ZooKeeperConnection Watcher:127.0.0.1:27978 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 271414 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 271415 T695 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 271419 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 271428 T704 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8a17663 name:ZooKeeperConnection Watcher:127.0.0.1:27978/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 271428 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 271428 T695 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 271433 T695 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 271441 T695 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 271444 T695 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 271447 T695 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 271447 T695 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 271457 T695 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 271458 T695 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 271562 T695 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 271563 T695 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 271574 T695 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 271574 T695 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 271578 T695 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 271578 T695 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 271582 T695 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 271582 T695 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 271586 T695 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 271586 T695 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 271589 T695 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 271590 T695 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 271593 T695 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 271594 T695 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 271814 T695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 271817 T695 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:60663
[junit4:junit4] 2> 271818 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 271819 T695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 271819 T695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368666933551
[junit4:junit4] 2> 271819 T695 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368666933551/solr.xml
[junit4:junit4] 2> 271820 T695 oasc.CoreContainer.<init> New CoreContainer 748718121
[junit4:junit4] 2> 271821 T695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368666933551/'
[junit4:junit4] 2> 271821 T695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368666933551/'
[junit4:junit4] 2> 271968 T695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 271968 T695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 271969 T695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 271969 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 271970 T695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 271970 T695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 271971 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 271971 T695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 271972 T695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 271972 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 271987 T695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 271987 T695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27978/solr
[junit4:junit4] 2> 271988 T695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 271989 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 271992 T715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77d2eed9 name:ZooKeeperConnection Watcher:127.0.0.1:27978 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 271992 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 271995 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 272005 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 272008 T717 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2aefa457 name:ZooKeeperConnection Watcher:127.0.0.1:27978/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 272008 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 272010 T695 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 272020 T695 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 272024 T695 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 272031 T695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60663_
[junit4:junit4] 2> 272033 T695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60663_
[junit4:junit4] 2> 272036 T695 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 272053 T695 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 272056 T695 oasc.Overseer.start Overseer (id=89696956137799683-127.0.0.1:60663_-n_0000000000) starting
[junit4:junit4] 2> 272067 T695 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 272079 T719 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 272079 T695 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 272082 T695 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 272089 T695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 272095 T718 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 272096 T720 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368666933551/collection1
[junit4:junit4] 2> 272096 T720 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 272097 T720 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 272097 T720 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 272098 T720 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368666933551/collection1/'
[junit4:junit4] 2> 272099 T720 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368666933551/collection1/lib/README' to classloader
[junit4:junit4] 2> 272099 T720 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368666933551/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 272163 T720 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 272226 T720 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 272328 T720 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 272335 T720 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 272772 T720 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 272775 T720 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 272777 T720 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 272781 T720 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 272807 T720 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 272808 T720 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368666933551/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/control/data/
[junit4:junit4] 2> 272808 T720 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@78497062
[junit4:junit4] 2> 272808 T720 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 272809 T720 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/control/data
[junit4:junit4] 2> 272809 T720 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/control/data/index/
[junit4:junit4] 2> 272809 T720 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 272810 T720 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/control/data/index
[junit4:junit4] 2> 272812 T720 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3fbebf70 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a8a0e4e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 272812 T720 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 272814 T720 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 272814 T720 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 272815 T720 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 272815 T720 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 272816 T720 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 272816 T720 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 272816 T720 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 272816 T720 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 272817 T720 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 272822 T720 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 272830 T720 oass.SolrIndexSearcher.<init> Opening Searcher@a7f7dd0 main
[junit4:junit4] 2> 272830 T720 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/control/data/tlog
[junit4:junit4] 2> 272831 T720 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 272831 T720 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 272833 T721 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a7f7dd0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 272834 T720 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 272834 T720 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 273600 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 273600 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60663_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60663"}
[junit4:junit4] 2> 273601 T718 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 273601 T718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 273616 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 273838 T720 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 273838 T720 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60663 collection:control_collection shard:shard1
[junit4:junit4] 2> 273839 T720 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 273859 T720 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 273862 T720 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 273862 T720 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 273863 T720 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60663/collection1/
[junit4:junit4] 2> 273863 T720 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 273863 T720 oasc.SyncStrategy.syncToMe http://127.0.0.1:60663/collection1/ has no replicas
[junit4:junit4] 2> 273863 T720 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60663/collection1/
[junit4:junit4] 2> 273863 T720 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 275121 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 275141 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 275198 T720 oasc.ZkController.register We are http://127.0.0.1:60663/collection1/ and leader is http://127.0.0.1:60663/collection1/
[junit4:junit4] 2> 275198 T720 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60663
[junit4:junit4] 2> 275198 T720 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 275198 T720 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 275198 T720 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 275201 T720 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 275202 T695 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 275202 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 275202 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 275212 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 275214 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 275216 T724 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7960123a name:ZooKeeperConnection Watcher:127.0.0.1:27978/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 275216 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 275218 T695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 275219 T695 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 275362 T695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 275365 T695 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15398
[junit4:junit4] 2> 275365 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 275366 T695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 275366 T695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368666937167
[junit4:junit4] 2> 275366 T695 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368666937167/solr.xml
[junit4:junit4] 2> 275367 T695 oasc.CoreContainer.<init> New CoreContainer 1413913341
[junit4:junit4] 2> 275367 T695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368666937167/'
[junit4:junit4] 2> 275367 T695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368666937167/'
[junit4:junit4] 2> 275444 T695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 275445 T695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 275445 T695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 275445 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 275446 T695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 275446 T695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 275446 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 275447 T695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 275447 T695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 275447 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 275458 T695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 275458 T695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27978/solr
[junit4:junit4] 2> 275459 T695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 275459 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 275463 T735 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40291a66 name:ZooKeeperConnection Watcher:127.0.0.1:27978 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 275463 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 275477 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 275485 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 275488 T737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79b20a36 name:ZooKeeperConnection Watcher:127.0.0.1:27978/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 275488 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 275499 T695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 276502 T695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15398_
[junit4:junit4] 2> 276531 T695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15398_
[junit4:junit4] 2> 276535 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 276536 T737 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 276536 T724 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 276537 T717 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 276543 T738 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368666937167/collection1
[junit4:junit4] 2> 276544 T738 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 276544 T738 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 276545 T738 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 276546 T738 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368666937167/collection1/'
[junit4:junit4] 2> 276547 T738 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368666937167/collection1/lib/README' to classloader
[junit4:junit4] 2> 276548 T738 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368666937167/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 276596 T738 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 276654 T738 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 276655 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 276656 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:60663__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60663_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60663"}
[junit4:junit4] 2> 276669 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 276669 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 276755 T738 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 276755 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 276763 T738 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 277299 T738 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 277302 T738 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 277304 T738 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 277308 T738 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 277329 T738 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 277329 T738 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368666937167/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty1/
[junit4:junit4] 2> 277330 T738 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@78497062
[junit4:junit4] 2> 277330 T738 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 277331 T738 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty1
[junit4:junit4] 2> 277331 T738 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty1/index/
[junit4:junit4] 2> 277331 T738 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 277331 T738 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty1/index
[junit4:junit4] 2> 277333 T738 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@76d0a86c lockFactory=org.apache.lucene.store.NativeFSLockFactory@270250a0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 277334 T738 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 277335 T738 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 277336 T738 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 277336 T738 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 277337 T738 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 277337 T738 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 277337 T738 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 277337 T738 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 277338 T738 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 277338 T738 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 277343 T738 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 277347 T738 oass.SolrIndexSearcher.<init> Opening Searcher@55c07d21 main
[junit4:junit4] 2> 277347 T738 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty1/tlog
[junit4:junit4] 2> 277348 T738 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 277348 T738 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 277351 T739 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@55c07d21 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 277352 T738 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 277352 T738 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 278173 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 278174 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15398_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15398"}
[junit4:junit4] 2> 278174 T718 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 278175 T718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 278192 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 278192 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 278192 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 278380 T738 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 278380 T738 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15398 collection:collection1 shard:shard1
[junit4:junit4] 2> 278381 T738 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 278399 T738 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 278410 T738 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 278411 T738 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 278411 T738 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:15398/collection1/
[junit4:junit4] 2> 278411 T738 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 278411 T738 oasc.SyncStrategy.syncToMe http://127.0.0.1:15398/collection1/ has no replicas
[junit4:junit4] 2> 278412 T738 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:15398/collection1/
[junit4:junit4] 2> 278412 T738 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 279697 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 279753 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 279753 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 279753 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 279796 T738 oasc.ZkController.register We are http://127.0.0.1:15398/collection1/ and leader is http://127.0.0.1:15398/collection1/
[junit4:junit4] 2> 279796 T738 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15398
[junit4:junit4] 2> 279796 T738 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 279796 T738 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 279797 T738 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 279799 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 279800 T695 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 279801 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 279801 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 280024 T695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 280027 T695 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28413
[junit4:junit4] 2> 280028 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 280028 T695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 280029 T695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368666941762
[junit4:junit4] 2> 280029 T695 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368666941762/solr.xml
[junit4:junit4] 2> 280029 T695 oasc.CoreContainer.<init> New CoreContainer 1683615091
[junit4:junit4] 2> 280030 T695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368666941762/'
[junit4:junit4] 2> 280031 T695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368666941762/'
[junit4:junit4] 2> 280128 T695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 280129 T695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 280129 T695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 280130 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 280130 T695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 280131 T695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 280131 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 280132 T695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 280132 T695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 280133 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 280147 T695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 280148 T695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27978/solr
[junit4:junit4] 2> 280148 T695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 280149 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 280157 T751 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13c9c980 name:ZooKeeperConnection Watcher:127.0.0.1:27978 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 280158 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 280172 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 280190 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 280195 T753 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d3cbbb1 name:ZooKeeperConnection Watcher:127.0.0.1:27978/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 280195 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 280210 T695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 281214 T695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28413_
[junit4:junit4] 2> 281259 T695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28413_
[junit4:junit4] 2> 281262 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 281263 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 281262 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 281263 T753 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 281263 T737 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 281264 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15398__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15398_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15398"}
[junit4:junit4] 2> 281264 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 281264 T724 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 281279 T754 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368666941762/collection1
[junit4:junit4] 2> 281280 T754 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 281281 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 281281 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 281281 T754 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 281281 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 281282 T754 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 281284 T754 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368666941762/collection1/'
[junit4:junit4] 2> 281286 T717 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 281286 T754 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368666941762/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 281286 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 281287 T754 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368666941762/collection1/lib/README' to classloader
[junit4:junit4] 2> 281335 T754 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 281392 T754 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 281494 T754 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 281503 T754 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 282053 T754 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 282057 T754 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 282059 T754 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 282065 T754 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 282095 T754 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 282096 T754 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368666941762/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty2/
[junit4:junit4] 2> 282096 T754 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@78497062
[junit4:junit4] 2> 282096 T754 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 282097 T754 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty2
[junit4:junit4] 2> 282098 T754 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty2/index/
[junit4:junit4] 2> 282098 T754 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 282099 T754 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty2/index
[junit4:junit4] 2> 282102 T754 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@383bd328 lockFactory=org.apache.lucene.store.NativeFSLockFactory@78aa6cbd),segFN=segments_1,generation=1}
[junit4:junit4] 2> 282102 T754 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 282105 T754 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 282105 T754 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 282106 T754 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 282106 T754 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 282107 T754 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 282107 T754 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 282107 T754 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 282108 T754 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 282108 T754 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 282116 T754 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 282122 T754 oass.SolrIndexSearcher.<init> Opening Searcher@48650005 main
[junit4:junit4] 2> 282122 T754 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty2/tlog
[junit4:junit4] 2> 282123 T754 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 282123 T754 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 282127 T755 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@48650005 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 282129 T754 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 282129 T754 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 282787 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 282788 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:28413_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:28413"}
[junit4:junit4] 2> 282788 T718 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 282788 T718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 282812 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 282813 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 282813 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 282812 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 283158 T754 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 283158 T754 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28413 collection:collection1 shard:shard2
[junit4:junit4] 2> 283159 T754 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 283177 T754 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 283186 T754 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 283186 T754 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 283186 T754 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:28413/collection1/
[junit4:junit4] 2> 283187 T754 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 283187 T754 oasc.SyncStrategy.syncToMe http://127.0.0.1:28413/collection1/ has no replicas
[junit4:junit4] 2> 283187 T754 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:28413/collection1/
[junit4:junit4] 2> 283187 T754 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 284327 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 284399 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 284399 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 284399 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 284399 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 284427 T754 oasc.ZkController.register We are http://127.0.0.1:28413/collection1/ and leader is http://127.0.0.1:28413/collection1/
[junit4:junit4] 2> 284427 T754 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28413
[junit4:junit4] 2> 284427 T754 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 284427 T754 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 284427 T754 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 284436 T754 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 284437 T695 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 284438 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 284439 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 284667 T695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 284671 T695 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:65373
[junit4:junit4] 2> 284672 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 284672 T695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 284672 T695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368666946401
[junit4:junit4] 2> 284673 T695 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368666946401/solr.xml
[junit4:junit4] 2> 284673 T695 oasc.CoreContainer.<init> New CoreContainer 1195846068
[junit4:junit4] 2> 284674 T695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368666946401/'
[junit4:junit4] 2> 284675 T695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368666946401/'
[junit4:junit4] 2> 284780 T695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 284780 T695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 284781 T695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 284781 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 284782 T695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 284782 T695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 284783 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 284783 T695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 284784 T695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 284784 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 284801 T695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 284802 T695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27978/solr
[junit4:junit4] 2> 284802 T695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 284803 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 284833 T767 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35a1028d name:ZooKeeperConnection Watcher:127.0.0.1:27978 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 284834 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 284849 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 284863 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 284865 T769 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5854278 name:ZooKeeperConnection Watcher:127.0.0.1:27978/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 284865 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 284878 T695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 285881 T695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65373_
[junit4:junit4] 2> 285916 T695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65373_
[junit4:junit4] 2> 285920 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 285920 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 285920 T737 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 285920 T769 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 285920 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 285920 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 285921 T724 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 285931 T717 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 285932 T753 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 285937 T770 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368666946401/collection1
[junit4:junit4] 2> 285937 T770 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 285938 T770 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 285938 T770 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 285939 T770 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368666946401/collection1/'
[junit4:junit4] 2> 285940 T770 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368666946401/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 285940 T770 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368666946401/collection1/lib/README' to classloader
[junit4:junit4] 2> 285963 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 285964 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:28413__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:28413_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:28413"}
[junit4:junit4] 2> 285973 T770 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 285977 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 285977 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 285977 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 285977 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 285977 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 286013 T770 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 286114 T770 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 286120 T770 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 286523 T770 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 286526 T770 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 286527 T770 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 286531 T770 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 286555 T770 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 286555 T770 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368666946401/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/
[junit4:junit4] 2> 286555 T770 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@78497062
[junit4:junit4] 2> 286556 T770 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 286557 T770 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3
[junit4:junit4] 2> 286557 T770 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/index/
[junit4:junit4] 2> 286557 T770 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 286557 T770 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/index
[junit4:junit4] 2> 286559 T770 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ac20952 lockFactory=org.apache.lucene.store.NativeFSLockFactory@26a9e0a6),segFN=segments_1,generation=1}
[junit4:junit4] 2> 286560 T770 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 286561 T770 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 286562 T770 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 286562 T770 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 286563 T770 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 286563 T770 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 286563 T770 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 286563 T770 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 286564 T770 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 286564 T770 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 286569 T770 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 286573 T770 oass.SolrIndexSearcher.<init> Opening Searcher@484cb430 main
[junit4:junit4] 2> 286573 T770 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/tlog
[junit4:junit4] 2> 286574 T770 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 286574 T770 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 286578 T771 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@484cb430 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 286580 T770 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 286580 T770 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 287482 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 287483 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:65373_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:65373"}
[junit4:junit4] 2> 287483 T718 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 287483 T718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 287525 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 287525 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 287525 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 287525 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 287525 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 287589 T770 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 287589 T770 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:65373 collection:collection1 shard:shard1
[junit4:junit4] 2> 287593 T770 oasc.ZkController.register We are http://127.0.0.1:65373/collection1/ and leader is http://127.0.0.1:15398/collection1/
[junit4:junit4] 2> 287593 T770 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:65373
[junit4:junit4] 2> 287594 T770 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 287594 T770 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@2270a34d url=http://127.0.0.1:65373/collection1 node=127.0.0.1:65373_ C29_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:65373_, base_url=http://127.0.0.1:65373}
[junit4:junit4] 2> 287594 T772 C29 P65373 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 287595 T770 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 287595 T772 C29 P65373 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 287596 T772 C29 P65373 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 287596 T772 C29 P65373 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 287597 T695 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 287597 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 287598 T772 C29 P65373 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 287598 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 287618 T727 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:65373__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 287943 T695 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 287947 T695 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:14398
[junit4:junit4] 2> 287948 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 287949 T695 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 287949 T695 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368666949564
[junit4:junit4] 2> 287950 T695 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368666949564/solr.xml
[junit4:junit4] 2> 287951 T695 oasc.CoreContainer.<init> New CoreContainer 1206956249
[junit4:junit4] 2> 287952 T695 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368666949564/'
[junit4:junit4] 2> 287953 T695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368666949564/'
[junit4:junit4] 2> 288108 T695 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 288109 T695 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 288110 T695 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 288110 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 288111 T695 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 288112 T695 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 288113 T695 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 288113 T695 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 288114 T695 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 288115 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 288138 T695 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 288139 T695 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27978/solr
[junit4:junit4] 2> 288140 T695 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 288142 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 288146 T784 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@308423ec name:ZooKeeperConnection Watcher:127.0.0.1:27978 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 288147 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 288163 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 288181 T695 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 288183 T786 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a0c9dd6 name:ZooKeeperConnection Watcher:127.0.0.1:27978/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 288184 T695 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 288196 T695 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 289030 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 289031 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:65373__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:65373_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:65373"}
[junit4:junit4] 2> 289048 T786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289048 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289048 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289048 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289048 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289048 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289201 T695 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:14398_
[junit4:junit4] 2> 289240 T695 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:14398_
[junit4:junit4] 2> 289305 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289305 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289305 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289305 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 289306 T786 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 289305 T737 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 289306 T786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 289306 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 289307 T769 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 289325 T717 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 289326 T753 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 289325 T724 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 289332 T787 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368666949564/collection1
[junit4:junit4] 2> 289332 T787 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 289333 T787 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 289333 T787 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 289334 T787 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368666949564/collection1/'
[junit4:junit4] 2> 289335 T787 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368666949564/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 289336 T787 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368666949564/collection1/lib/README' to classloader
[junit4:junit4] 2> 289369 T787 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 289407 T787 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 289509 T787 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 289515 T787 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 289621 T727 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:65373__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 289621 T727 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:65373__collection1&state=recovering&nodeName=127.0.0.1:65373_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 290080 T787 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 290084 T787 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 290087 T787 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 290092 T787 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 290123 T787 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 290123 T787 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368666949564/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/
[junit4:junit4] 2> 290123 T787 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@78497062
[junit4:junit4] 2> 290124 T787 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 290124 T787 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4
[junit4:junit4] 2> 290125 T787 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/index/
[junit4:junit4] 2> 290125 T787 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 290126 T787 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/index
[junit4:junit4] 2> 290128 T787 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@48552f85 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1add52d8),segFN=segments_1,generation=1}
[junit4:junit4] 2> 290129 T787 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 290131 T787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 290132 T787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 290132 T787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 290133 T787 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 290134 T787 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 290134 T787 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 290134 T787 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 290135 T787 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 290135 T787 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 290142 T787 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 290149 T787 oass.SolrIndexSearcher.<init> Opening Searcher@3633794f main
[junit4:junit4] 2> 290149 T787 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/tlog
[junit4:junit4] 2> 290150 T787 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 290150 T787 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 290154 T788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3633794f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 290156 T787 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 290156 T787 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 290554 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 290554 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:14398_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:14398"}
[junit4:junit4] 2> 290555 T718 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 290555 T718 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 290574 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 290574 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 290574 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 290574 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 290574 T786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 290574 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 291159 T787 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 291159 T787 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:14398 collection:collection1 shard:shard2
[junit4:junit4] 2> 291164 T787 oasc.ZkController.register We are http://127.0.0.1:14398/collection1/ and leader is http://127.0.0.1:28413/collection1/
[junit4:junit4] 2> 291164 T787 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:14398
[junit4:junit4] 2> 291165 T787 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 291165 T787 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@41bfa8c8 url=http://127.0.0.1:14398/collection1 node=127.0.0.1:14398_ C30_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:14398_, base_url=http://127.0.0.1:14398}
[junit4:junit4] 2> 291165 T789 C30 P14398 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 291166 T787 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 291166 T789 C30 P14398 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 291167 T789 C30 P14398 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 291167 T789 C30 P14398 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 291167 T695 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 2> 291168 T695 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 291169 T695 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 291169 T789 C30 P14398 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 291192 T695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 291194 T743 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:14398__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 291194 T695 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 291195 T695 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 291195 T695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C29_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:65373_, base_url=http://127.0.0.1:65373}
[junit4:junit4] 2> 291623 T772 C29 P65373 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:15398/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 291623 T772 C29 P65373 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:65373 START replicas=[http://127.0.0.1:15398/collection1/] nUpdates=100
[junit4:junit4] 2> 291624 T772 C29 P65373 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 291624 T772 C29 P65373 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 291624 T772 C29 P65373 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 291624 T772 C29 P65373 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 291624 T772 C29 P65373 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 291625 T772 C29 P65373 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:15398/collection1/. core=collection1
[junit4:junit4] 2> 291625 T772 C29 P65373 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@7cbd9f51 url=http://127.0.0.1:15398/collection1 node=127.0.0.1:15398_ C31_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:15398_, base_url=http://127.0.0.1:15398, leader=true}
[junit4:junit4] 2> 291631 T728 C31 P15398 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 291634 T729 C31 P15398 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 291637 T729 C31 P15398 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@76d0a86c lockFactory=org.apache.lucene.store.NativeFSLockFactory@270250a0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 291637 T729 C31 P15398 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 291638 T729 C31 P15398 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@76d0a86c lockFactory=org.apache.lucene.store.NativeFSLockFactory@270250a0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@76d0a86c lockFactory=org.apache.lucene.store.NativeFSLockFactory@270250a0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 291638 T729 C31 P15398 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 291639 T729 C31 P15398 oass.SolrIndexSearcher.<init> Opening Searcher@794078f1 realtime
[junit4:junit4] 2> 291639 T729 C31 P15398 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 291639 T729 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 291640 T772 C29 P65373 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 291641 T772 C29 P65373 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 291643 T730 C31 P15398 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 291643 T730 C31 P15398 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 291643 T772 C29 P65373 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 291643 T772 C29 P65373 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 291644 T772 C29 P65373 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 291646 T730 C31 P15398 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 291647 T772 C29 P65373 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 291648 T772 C29 P65373 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/index.20130516041553594
[junit4:junit4] 2> 291648 T772 C29 P65373 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@19e502d8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d01739b) fullCopy=false
[junit4:junit4] 2> 291651 T730 C31 P15398 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 291652 T772 C29 P65373 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 291653 T772 C29 P65373 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 291653 T772 C29 P65373 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 291654 T772 C29 P65373 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ac20952 lockFactory=org.apache.lucene.store.NativeFSLockFactory@26a9e0a6),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ac20952 lockFactory=org.apache.lucene.store.NativeFSLockFactory@26a9e0a6),segFN=segments_2,generation=2}
[junit4:junit4] 2> 291655 T772 C29 P65373 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 291655 T772 C29 P65373 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 291655 T772 C29 P65373 oass.SolrIndexSearcher.<init> Opening Searcher@2ac1105f main
[junit4:junit4] 2> 291657 T771 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2ac1105f main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 291657 T772 C29 P65373 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/index.20130516041553594 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/index.20130516041553594;done=true>>]
[junit4:junit4] 2> 291658 T772 C29 P65373 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/index.20130516041553594
[junit4:junit4] 2> 291658 T772 C29 P65373 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty3/index.20130516041553594
[junit4:junit4] 2> 291658 T772 C29 P65373 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 291658 T772 C29 P65373 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 291658 T772 C29 P65373 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 291658 T772 C29 P65373 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 291660 T772 C29 P65373 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 292080 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 292080 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:14398__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:14398_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:14398"}
[junit4:junit4] 2> 292084 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:65373__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:65373_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:65373"}
[junit4:junit4] 2> 292096 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 292096 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 292096 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 292096 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 292096 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 292096 T786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 292195 T743 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:14398__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 292195 T743 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:14398__collection1&state=recovering&nodeName=127.0.0.1:14398_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 292197 T695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 293200 T695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C30_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:14398_, base_url=http://127.0.0.1:14398}
[junit4:junit4] 2> 294197 T789 C30 P14398 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:28413/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 294198 T789 C30 P14398 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:14398 START replicas=[http://127.0.0.1:28413/collection1/] nUpdates=100
[junit4:junit4] 2> 294198 T789 C30 P14398 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 294198 T789 C30 P14398 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 294199 T789 C30 P14398 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 294199 T789 C30 P14398 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 294199 T789 C30 P14398 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 294199 T789 C30 P14398 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:28413/collection1/. core=collection1
[junit4:junit4] 2> 294200 T789 C30 P14398 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 294202 T695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C32 name=collection1 org.apache.solr.core.SolrCore@64bbe1ee url=http://127.0.0.1:28413/collection1 node=127.0.0.1:28413_ C32_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:28413_, base_url=http://127.0.0.1:28413, leader=true}
[junit4:junit4] 2> 294208 T744 C32 P28413 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 294213 T745 C32 P28413 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 294215 T745 C32 P28413 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@383bd328 lockFactory=org.apache.lucene.store.NativeFSLockFactory@78aa6cbd),segFN=segments_1,generation=1}
[junit4:junit4] 2> 294215 T745 C32 P28413 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 294216 T745 C32 P28413 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@383bd328 lockFactory=org.apache.lucene.store.NativeFSLockFactory@78aa6cbd),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@383bd328 lockFactory=org.apache.lucene.store.NativeFSLockFactory@78aa6cbd),segFN=segments_2,generation=2}
[junit4:junit4] 2> 294217 T745 C32 P28413 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 294217 T745 C32 P28413 oass.SolrIndexSearcher.<init> Opening Searcher@106d0c0 realtime
[junit4:junit4] 2> 294218 T745 C32 P28413 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 294218 T745 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 294219 T789 C30 P14398 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 294219 T789 C30 P14398 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 294221 T746 C32 P28413 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 294221 T746 C32 P28413 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 294222 T789 C30 P14398 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 294222 T789 C30 P14398 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 294222 T789 C30 P14398 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 294226 T746 C32 P28413 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 294227 T789 C30 P14398 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 294228 T789 C30 P14398 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/index.20130516041556175
[junit4:junit4] 2> 294229 T789 C30 P14398 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@7fd021bc lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c0e69d2) fullCopy=false
[junit4:junit4] 2> 294231 T746 C32 P28413 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 294233 T789 C30 P14398 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 294234 T789 C30 P14398 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 294234 T789 C30 P14398 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 294236 T789 C30 P14398 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@48552f85 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1add52d8),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@48552f85 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1add52d8),segFN=segments_2,generation=2}
[junit4:junit4] 2> 294236 T789 C30 P14398 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 294237 T789 C30 P14398 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 294237 T789 C30 P14398 oass.SolrIndexSearcher.<init> Opening Searcher@b6edc37 main
[junit4:junit4] 2> 294238 T788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b6edc37 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 294239 T789 C30 P14398 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/index.20130516041556175 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/index.20130516041556175;done=true>>]
[junit4:junit4] 2> 294239 T789 C30 P14398 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/index.20130516041556175
[junit4:junit4] 2> 294239 T789 C30 P14398 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368666933249/jetty4/index.20130516041556175
[junit4:junit4] 2> 294244 T789 C30 P14398 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 294244 T789 C30 P14398 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 294245 T789 C30 P14398 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 294245 T789 C30 P14398 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 294252 T789 C30 P14398 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 295113 T718 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 295114 T718 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:14398__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:14398_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:14398"}
[junit4:junit4] 2> 295136 T786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 295136 T717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 295136 T769 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 295136 T753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 295136 T724 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 295136 T737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 295204 T695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 295206 T695 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C33 name=collection1 org.apache.solr.core.SolrCore@6ffff277 url=http://127.0.0.1:60663/collection1 node=127.0.0.1:60663_ C33_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:60663_, base_url=http://127.0.0.1:60663, leader=true}
[junit4:junit4] 2> 295214 T707 C33 P60663 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 295217 T707 C33 P60663 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3fbebf70 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a8a0e4e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 295218 T707 C33 P60663 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 295219 T707 C33 P60663 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3fbebf70 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a8a0e4e),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3fbebf70 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a8a0e4e),segFN=segments_2,generation=2}
[junit4:junit4] 2> 295219 T707 C33 P60663 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 295220 T707 C33 P60663 oass.SolrIndexSearcher.<init> Opening Searcher@6240dc93 main
[junit4:junit4] 2> 295220 T707 C33 P60663 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 295221 T721 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6240dc93 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 295221 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 295234 T747 C32 P28413 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 295239 T747 C32 P28413 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@383bd328 lockFactory=org.apache.lucene.store.NativeFSLockFactory@78aa6cbd),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@383bd328 lockFactory=org.apache.lucene.store.NativeFSLockFactory@78aa6cbd),segFN=segments_3,generation=3}
[junit4:junit4] 2> 295240 T747 C32 P28413 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 295240 T747 C32 P28413 oass.SolrIndexSearcher.<init> Opening Searcher@23bff716 main
[junit4:junit4] 2> 295241 T747 C32 P28413 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 295242 T755 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23bff716 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 295242 T747 C32 P28413 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:15398/collection1/, StdNode: http://127.0.0.1:65373/collection1/, StdNode: http://127.0.0.1:14398/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 295252 T731 C31 P15398 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 295254 T731 C31 P15398 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@76d0a86c lockFactory=org.apache.lucene.store.NativeFSLockFactory@270250a0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@76d0a86c lockFactory=org.apache.lucene.store.NativeFSLockFactory@270250a0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 295254 T731 C31 P15398 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 295255 T731 C31 P15398 oass.SolrIndexSearcher.<init> Opening Searcher@5f17c550 main
[junit4:junit4] 2> 295256 T731 C31 P15398 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 295256 T739 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f17c550 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 295257 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> C30_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:14398_, base_url=http://127.0.0.1:14398}
[junit4:junit4] 2> 295259 T776 C30 P14398 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 295260 T776 C30 P14398 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@48552f85 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1add52d8),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@48552f85 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1add52d8),segFN=segments_3,generation=3}
[junit4:junit4] 2> 295261 T776 C30 P14398 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> C29_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:65373_, base_url=http://127.0.0.1:65373}
[junit4:junit4] 2> 295261 T759 C29 P65373 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 295261 T776 C30 P14398 oass.SolrIndexSearcher.<init> Opening Searcher@7c9c8a7b main
[junit4:junit4] 2> 295262 T759 C29 P65373 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ac20952 lockFactory=org.apache.lucene.store.NativeFSLockFactory@26a9e0a6),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ac20952 lockFactory=org.apache.lucene.store.NativeFSLockFactory@26a9e0a6),segFN=segments_3,generation=3}
[junit4:junit4] 2> 295263 T759 C29 P65373 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 295262 T776 C30 P14398 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 295263 T759 C29 P65373 oass.SolrIndexSearcher.<init> Opening Searcher@4584c87f main
[junit4:junit4] 2> 295263 T788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c9c8a7b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 295264 T759 C29 P65373 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 295264 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 295264 T771 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4584c87f main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 295265 T759 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4] 2> 295266 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 32
[junit4:junit4] 2> 295267 T695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 295271 T732 C31 P15398 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 295274 T760 C29 P65373 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 295277 T748 C32 P28413 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 295279 T777 C30 P14398 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 297290 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435151325445750784)} 0 6
[junit4:junit4] 2> 297298 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1435151325452042240&update.from=http://127.0.0.1:28413/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435151325452042240)} 0 1
[junit4:junit4] 2> 297303 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1435151325457285120&update.from=http://127.0.0.1:15398/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435151325457285120)} 0 2
[junit4:junit4] 2> 297305 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1435151325457285120)} 0 7
[junit4:junit4] 2> 297305 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435151325452042240)} 0 12
[junit4:junit4] 2> 297309 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1435151325468819456)]} 0 1
[junit4:junit4] 2> 297317 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1435151325471965184)]} 0 2
[junit4:junit4] 2> 297318 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1435151325471965184)]} 0 6
[junit4:junit4] 2> 297322 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1435151325482450944)]} 0 1
[junit4:junit4] 2> 297332 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1435151325488742400)]} 0 1
[junit4:junit4] 2> 297333 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1435151325488742400)]} 0 4
[junit4:junit4] 2> 297333 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1]} 0 8
[junit4:junit4] 2> 297337 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1435151325497131008)]} 0 1
[junit4:junit4] 2> 297346 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1435151325503422464)]} 0 1
[junit4:junit4] 2> 297346 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[2 (1435151325503422464)]} 0 4
[junit4:junit4] 2> 297347 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2]} 0 7
[junit4:junit4] 2> 297351 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1435151325512859648)]} 0 1
[junit4:junit4] 2> 297358 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1435151325516005376)]} 0 1
[junit4:junit4] 2> 297358 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1435151325516005376)]} 0 4
[junit4:junit4] 2> 297362 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1435151325523345408)]} 0 1
[junit4:junit4] 2> 297368 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1435151325528588288)]} 0 0
[junit4:junit4] 2> 297369 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1435151325528588288)]} 0 4
[junit4:junit4] 2> 297373 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1435151325534879744)]} 0 1
[junit4:junit4] 2> 297380 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1435151325539074048)]} 0 1
[junit4:junit4] 2> 297381 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1435151325539074048)]} 0 5
[junit4:junit4] 2> 297384 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1435151325546414080)]} 0 1
[junit4:junit4] 2> 297394 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1435151325553754112)]} 0 1
[junit4:junit4] 2> 297394 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[6 (1435151325553754112)]} 0 4
[junit4:junit4] 2> 297395 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6]} 0 8
[junit4:junit4] 2> 297399 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1435151325562142720)]} 0 1
[junit4:junit4] 2> 297407 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1435151325568434176)]} 0 0
[junit4:junit4] 2> 297408 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[7 (1435151325568434176)]} 0 4
[junit4:junit4] 2> 297409 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7]} 0 8
[junit4:junit4] 2> 297413 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1435151325576822784)]} 0 1
[junit4:junit4] 2> 297435 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1435151325594648576)]} 0 1
[junit4:junit4] 2> 297436 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[8 (1435151325594648576)]} 0 7
[junit4:junit4] 2> 297437 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8]} 0 22
[junit4:junit4] 2> 297441 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1435151325606182912)]} 0 1
[junit4:junit4] 2> 297450 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1435151325613522944)]} 0 0
[junit4:junit4] 2> 297451 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[9 (1435151325613522944)]} 0 4
[junit4:junit4] 2> 297452 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9]} 0 8
[junit4:junit4] 2> 297456 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1435151325621911552)]} 0 1
[junit4:junit4] 2> 297462 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1435151325625057280)]} 0 1
[junit4:junit4] 2> 297463 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1435151325625057280)]} 0 5
[junit4:junit4] 2> 297466 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1435151325633445888)]} 0 0
[junit4:junit4] 2> 297476 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1435151325639737344)]} 0 1
[junit4:junit4] 2> 297477 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[11 (1435151325639737344)]} 0 5
[junit4:junit4] 2> 297477 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11]} 0 8
[junit4:junit4] 2> 297481 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1435151325649174528)]} 0 1
[junit4:junit4] 2> 297491 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1435151325655465984)]} 0 0
[junit4:junit4] 2> 297492 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[12 (1435151325655465984)]} 0 5
[junit4:junit4] 2> 297493 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12]} 0 9
[junit4:junit4] 2> 297497 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1435151325664903168)]} 0 1
[junit4:junit4] 2> 297507 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1435151325672243200)]} 0 1
[junit4:junit4] 2> 297508 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[13 (1435151325672243200)]} 0 5
[junit4:junit4] 2> 297509 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13]} 0 8
[junit4:junit4] 2> 297512 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1435151325680631808)]} 0 1
[junit4:junit4] 2> 297521 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1435151325686923264)]} 0 1
[junit4:junit4] 2> 297522 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[14 (1435151325686923264)]} 0 5
[junit4:junit4] 2> 297523 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14]} 0 8
[junit4:junit4] 2> 297527 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1435151325696360448)]} 0 1
[junit4:junit4] 2> 297533 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1435151325699506176)]} 0 1
[junit4:junit4] 2> 297534 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1435151325699506176)]} 0 5
[junit4:junit4] 2> 297537 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1435151325706846208)]} 0 1
[junit4:junit4] 2> 297547 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1435151325714186240)]} 0 1
[junit4:junit4] 2> 297548 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[16 (1435151325714186240)]} 0 5
[junit4:junit4] 2> 297549 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16]} 0 9
[junit4:junit4] 2> 297553 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1435151325723623424)]} 0 1
[junit4:junit4] 2> 297558 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1435151325726769152)]} 0 0
[junit4:junit4] 2> 297559 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1435151325726769152)]} 0 4
[junit4:junit4] 2> 297563 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1435151325734109184)]} 0 1
[junit4:junit4] 2> 297573 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1435151325741449216)]} 0 1
[junit4:junit4] 2> 297573 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[18 (1435151325741449216)]} 0 4
[junit4:junit4] 2> 297574 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18]} 0 8
[junit4:junit4] 2> 297577 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1435151325749837824)]} 0 0
[junit4:junit4] 2> 297586 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1435151325756129280)]} 0 0
[junit4:junit4] 2> 297587 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[19 (1435151325756129280)]} 0 4
[junit4:junit4] 2> 297588 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19]} 0 8
[junit4:junit4] 2> 297591 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1435151325763469312)]} 0 1
[junit4:junit4] 2> 297599 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1435151325768712192)]} 0 1
[junit4:junit4] 2> 297600 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[20 (1435151325768712192)]} 0 5
[junit4:junit4] 2> 297601 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20]} 0 8
[junit4:junit4] 2> 297605 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1435151325778149376)]} 0 1
[junit4:junit4] 2> 297614 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1435151325784440832)]} 0 0
[junit4:junit4] 2> 297614 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[21 (1435151325784440832)]} 0 4
[junit4:junit4] 2> 297615 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21]} 0 8
[junit4:junit4] 2> 297619 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1435151325792829440)]} 0 1
[junit4:junit4] 2> 297625 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1435151325795975168)]} 0 1
[junit4:junit4] 2> 297626 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1435151325795975168)]} 0 5
[junit4:junit4] 2> 297630 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1435151325804363776)]} 0 1
[junit4:junit4] 2> 297636 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1435151325807509504)]} 0 1
[junit4:junit4] 2> 297636 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1435151325807509504)]} 0 4
[junit4:junit4] 2> 297639 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1435151325814849536)]} 0 0
[junit4:junit4] 2> 297649 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1435151325821140992)]} 0 1
[junit4:junit4] 2> 297650 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[24 (1435151325821140992)]} 0 5
[junit4:junit4] 2> 297651 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24]} 0 9
[junit4:junit4] 2> 297654 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1435151325829529600)]} 0 1
[junit4:junit4] 2> 297659 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1435151325832675328)]} 0 0
[junit4:junit4] 2> 297660 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1435151325832675328)]} 0 4
[junit4:junit4] 2> 297663 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1435151325840015360)]} 0 0
[junit4:junit4] 2> 297669 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1435151325843161088)]} 0 0
[junit4:junit4] 2> 297670 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1435151325843161088)]} 0 4
[junit4:junit4] 2> 297673 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1435151325850501120)]} 0 0
[junit4:junit4] 2> 297682 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1435151325855744000)]} 0 1
[junit4:junit4] 2> 297682 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[27 (1435151325855744000)]} 0 4
[junit4:junit4] 2> 297683 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27]} 0 7
[junit4:junit4] 2> 297686 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1435151325864132608)]} 0 0
[junit4:junit4] 2> 297692 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1435151325867278336)]} 0 1
[junit4:junit4] 2> 297692 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1435151325867278336)]} 0 3
[junit4:junit4] 2> 297695 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1435151325873569792)]} 0 0
[junit4:junit4] 2> 297703 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1435151325878812672)]} 0 0
[junit4:junit4] 2> 297704 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[29 (1435151325878812672)]} 0 4
[junit4:junit4] 2> 297704 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29]} 0 6
[junit4:junit4] 2> 297707 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1435151325886152704)]} 0 0
[junit4:junit4] 2> 297713 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1435151325889298432)]} 0 0
[junit4:junit4] 2> 297714 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1435151325889298432)]} 0 4
[junit4:junit4] 2> 297717 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1435151325896638464)]} 0 0
[junit4:junit4] 2> 297724 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1435151325901881344)]} 0 0
[junit4:junit4] 2> 297725 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[31 (1435151325901881344)]} 0 3
[junit4:junit4] 2> 297726 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31]} 0 6
[junit4:junit4] 2> 297728 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1435151325908172800)]} 0 0
[junit4:junit4] 2> 297738 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1435151325914464256)]} 0 1
[junit4:junit4] 2> 297738 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[32 (1435151325914464256)]} 0 4
[junit4:junit4] 2> 297739 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32]} 0 8
[junit4:junit4] 2> 297742 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1435151325922852864)]} 0 0
[junit4:junit4] 2> 297748 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1435151325925998592)]} 0 0
[junit4:junit4] 2> 297749 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1435151325925998592)]} 0 4
[junit4:junit4] 2> 297752 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1435151325933338624)]} 0 0
[junit4:junit4] 2> 297759 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1435151325936484352)]} 0 1
[junit4:junit4] 2> 297759 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1435151325936484352)]} 0 4
[junit4:junit4] 2> 297762 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1435151325943824384)]} 0 0
[junit4:junit4] 2> 297768 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1435151325946970112)]} 0 1
[junit4:junit4] 2> 297768 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1435151325946970112)]} 0 3
[junit4:junit4] 2> 297771 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1435151325953261568)]} 0 0
[junit4:junit4] 2> 297780 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1435151325958504448)]} 0 1
[junit4:junit4] 2> 297781 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[36 (1435151325958504448)]} 0 5
[junit4:junit4] 2> 297782 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36]} 0 8
[junit4:junit4] 2> 297785 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1435151325967941632)]} 0 0
[junit4:junit4] 2> 297791 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1435151325971087360)]} 0 0
[junit4:junit4] 2> 297792 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1435151325971087360)]} 0 4
[junit4:junit4] 2> 297794 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1435151325977378816)]} 0 0
[junit4:junit4] 2> 297801 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1435151325980524544)]} 0 1
[junit4:junit4] 2> 297802 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1435151325980524544)]} 0 5
[junit4:junit4] 2> 297806 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1435151325988913152)]} 0 1
[junit4:junit4] 2> 297811 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1435151325992058880)]} 0 0
[junit4:junit4] 2> 297812 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1435151325992058880)]} 0 4
[junit4:junit4] 2> 297815 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1435151325999398912)]} 0 0
[junit4:junit4] 2> 297824 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1435151326004641792)]} 0 1
[junit4:junit4] 2> 297825 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[40 (1435151326004641792)]} 0 5
[junit4:junit4] 2> 297825 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40]} 0 7
[junit4:junit4] 2> 297828 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1435151326013030400)]} 0 0
[junit4:junit4] 2> 297837 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1435151326019321856)]} 0 0
[junit4:junit4] 2> 297838 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[41 (1435151326019321856)]} 0 4
[junit4:junit4] 2> 297839 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41]} 0 8
[junit4:junit4] 2> 297842 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1435151326026661888)]} 0 1
[junit4:junit4] 2> 297851 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1435151326032953344)]} 0 1
[junit4:junit4] 2> 297851 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[42 (1435151326032953344)]} 0 4
[junit4:junit4] 2> 297852 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42]} 0 8
[junit4:junit4] 2> 297854 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1435151326040293376)]} 0 0
[junit4:junit4] 2> 297861 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1435151326043439104)]} 0 1
[junit4:junit4] 2> 297861 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1435151326043439104)]} 0 4
[junit4:junit4] 2> 297865 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1435151326050779136)]} 0 1
[junit4:junit4] 2> 297870 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1435151326053924864)]} 0 0
[junit4:junit4] 2> 297871 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1435151326053924864)]} 0 4
[junit4:junit4] 2> 297874 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1435151326061264896)]} 0 0
[junit4:junit4] 2> 297883 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1435151326067556352)]} 0 0
[junit4:junit4] 2> 297884 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[45 (1435151326067556352)]} 0 4
[junit4:junit4] 2> 297885 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45]} 0 8
[junit4:junit4] 2> 297888 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1435151326075944960)]} 0 0
[junit4:junit4] 2> 297896 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1435151326081187840)]} 0 0
[junit4:junit4] 2> 297897 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[46 (1435151326081187840)]} 0 4
[junit4:junit4] 2> 297898 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46]} 0 7
[junit4:junit4] 2> 297900 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1435151326088527872)]} 0 0
[junit4:junit4] 2> 297906 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1435151326091673600)]} 0 0
[junit4:junit4] 2> 297907 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1435151326091673600)]} 0 4
[junit4:junit4] 2> 297910 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1435151326099013632)]} 0 0
[junit4:junit4] 2> 297918 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1435151326104256512)]} 0 0
[junit4:junit4] 2> 297919 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[48 (1435151326104256512)]} 0 4
[junit4:junit4] 2> 297920 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48]} 0 7
[junit4:junit4] 2> 297923 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1435151326111596544)]} 0 1
[junit4:junit4] 2> 297933 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1435151326118936576)]} 0 1
[junit4:junit4] 2> 297933 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[49 (1435151326118936576)]} 0 4
[junit4:junit4] 2> 297934 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49]} 0 8
[junit4:junit4] 2> 297938 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1435151326127325184)]} 0 1
[junit4:junit4] 2> 297946 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1435151326132568064)]} 0 1
[junit4:junit4] 2> 297946 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1435151326132568064)]} 0 5
[junit4:junit4] 2> 297950 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1435151326139908096)]} 0 1
[junit4:junit4] 2> 297959 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1435151326146199552)]} 0 1
[junit4:junit4] 2> 297959 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[51 (1435151326146199552)]} 0 4
[junit4:junit4] 2> 297960 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51]} 0 8
[junit4:junit4] 2> 297964 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1435151326154588160)]} 0 1
[junit4:junit4] 2> 297970 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1435151326158782464)]} 0 0
[junit4:junit4] 2> 297971 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1435151326158782464)]} 0 4
[junit4:junit4] 2> 297975 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1435151326166122496)]} 0 1
[junit4:junit4] 2> 297984 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1435151326172413952)]} 0 1
[junit4:junit4] 2> 297984 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[53 (1435151326172413952)]} 0 4
[junit4:junit4] 2> 297985 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53]} 0 8
[junit4:junit4] 2> 297989 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1435151326180802560)]} 0 1
[junit4:junit4] 2> 297998 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1435151326187094016)]} 0 1
[junit4:junit4] 2> 297999 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[54 (1435151326187094016)]} 0 5
[junit4:junit4] 2> 297999 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54]} 0 8
[junit4:junit4] 2> 298003 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1435151326195482624)]} 0 1
[junit4:junit4] 2> 298012 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1435151326201774080)]} 0 1
[junit4:junit4] 2> 298013 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[55 (1435151326201774080)]} 0 5
[junit4:junit4] 2> 298013 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55]} 0 8
[junit4:junit4] 2> 298017 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1435151326210162688)]} 0 1
[junit4:junit4] 2> 298026 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1435151326216454144)]} 0 1
[junit4:junit4] 2> 298027 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[56 (1435151326216454144)]} 0 5
[junit4:junit4] 2> 298027 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56]} 0 8
[junit4:junit4] 2> 298031 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1435151326224842752)]} 0 1
[junit4:junit4] 2> 298037 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1435151326227988480)]} 0 1
[junit4:junit4] 2> 298038 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1435151326227988480)]} 0 5
[junit4:junit4] 2> 298041 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1435151326235328512)]} 0 1
[junit4:junit4] 2> 298047 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1435151326238474240)]} 0 1
[junit4:junit4] 2> 298048 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1435151326238474240)]} 0 5
[junit4:junit4] 2> 298052 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1435151326246862848)]} 0 1
[junit4:junit4] 2> 298059 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1435151326251057152)]} 0 1
[junit4:junit4] 2> 298060 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1435151326251057152)]} 0 5
[junit4:junit4] 2> 298064 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1435151326259445760)]} 0 1
[junit4:junit4] 2> 298073 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1435151326266785792)]} 0 1
[junit4:junit4] 2> 298074 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[60 (1435151326266785792)]} 0 5
[junit4:junit4] 2> 298074 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60]} 0 8
[junit4:junit4] 2> 298077 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1435151326274125824)]} 0 0
[junit4:junit4] 2> 298083 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[61 (1435151326277271552)]} 0 0
[junit4:junit4] 2> 298084 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1435151326277271552)]} 0 4
[junit4:junit4] 2> 298088 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1435151326284611584)]} 0 1
[junit4:junit4] 2> 298098 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[62 (1435151326291951616)]} 0 1
[junit4:junit4] 2> 298099 T749 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[62 (1435151326291951616)]} 0 5
[junit4:junit4] 2> 298100 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62]} 0 9
[junit4:junit4] 2> 298103 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1435151326300340224)]} 0 1
[junit4:junit4] 2> 298113 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[63 (1435151326307680256)]} 0 1
[junit4:junit4] 2> 298114 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[63 (1435151326307680256)]} 0 5
[junit4:junit4] 2> 298114 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63]} 0 8
[junit4:junit4] 2> 298118 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[64 (1435151326316068864)]} 0 1
[junit4:junit4] 2> 298125 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[64 (1435151326321311744)]} 0 1
[junit4:junit4] 2> 298125 T733 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[64 (1435151326321311744)]} 0 4
[junit4:junit4] 2> 298129 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[65 (1435151326327603200)]} 0 1
[junit4:junit4] 2> 298138 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[65 (1435151326333894656)]} 0 1
[junit4:junit4] 2> 298139 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[65 (1435151326333894656)]} 0 5
[junit4:junit4] 2> 298139 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[65]} 0 8
[junit4:junit4] 2> 298143 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[66 (1435151326342283264)]} 0 1
[junit4:junit4] 2> 298152 T761 C29 P65373 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:15398/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[66 (1435151326348574720)]} 0 1
[junit4:junit4] 2> 298153 T731 C31 P15398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[66 (1435151326348574720)]} 0 5
[junit4:junit4] 2> 298153 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[66]} 0 8
[junit4:junit4] 2> 298157 T707 C33 P60663 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[67 (1435151326356963328)]} 0 1
[junit4:junit4] 2> 298163 T776 C30 P14398 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:28413/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[67 (1435151326360109056)]} 0 1
[junit4:junit4] 2> 298163 T747 C32 P28413 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=
[...truncated too long message...]
L, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, SCHEMA-SYNONYM-TOKENIZER.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-REST-LUCENE-MATCH-VERSION.XML, SCHEMA-DOCVALUESFACETING.XML, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, XSLT, SOLRCONFIG-NATIVELOCK.XML, SOLRCONFIG-MUTABLE-MANAGED-SCHEMA.XML, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, BAD-SCHEMA-SWEETSPOT-BOTH-TF.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, BAD-CURRENCY.XML, SOLRCONFIG_PERF.XML, BAD-SCHEMA-CURRENCY-FT-BOGUS-DEFAULT-CODE.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4] 2> 928070 T1197 oass.SolrIndexSearcher.<init> Opening Searcher@64a8bd73 main
[junit4:junit4] 2> 928070 T1197 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 928070 T1197 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 928071 T1197 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4] 2> 928088 T1197 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4] 2> 928140 T1198 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@64a8bd73 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 928143 T1197 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 928148 T1197 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4] 2> 928156 T1197 oas.SolrTestCaseJ4.setUp ###Starting testTwoRules
[junit4:junit4] 2> 928156 T1197 oas.SolrTestCaseJ4.tearDown ###Ending testTwoRules
[junit4:junit4] 2> 928159 T1197 oas.SolrTestCaseJ4.setUp ###Starting testBoostGood
[junit4:junit4] 2> 928160 T1197 oas.SolrTestCaseJ4.tearDown ###Ending testBoostGood
[junit4:junit4] 2> 928163 T1197 oas.SolrTestCaseJ4.setUp ###Starting testNoBoost
[junit4:junit4] 2> 928163 T1197 oas.SolrTestCaseJ4.tearDown ###Ending testNoBoost
[junit4:junit4] 2> 928167 T1197 oas.SolrTestCaseJ4.setUp ###Starting testDeboostOld
[junit4:junit4] 2> 928167 T1197 oas.SolrTestCaseJ4.tearDown ###Ending testDeboostOld
[junit4:junit4] 2> 928168 T1197 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 928168 T1197 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=89463863
[junit4:junit4] 2> 928168 T1197 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@16af619d
[junit4:junit4] 2> 928176 T1197 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 928177 T1197 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 928177 T1197 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 928177 T1197 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 928177 T1197 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 928178 T1197 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 928178 T1197 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368667589229/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368667589229/index;done=false>>]
[junit4:junit4] 2> 928178 T1197 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368667589229/index
[junit4:junit4] 2> 928178 T1197 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368667589229 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368667589229;done=false>>]
[junit4:junit4] 2> 928179 T1197 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RegexBoostProcessorTest-1368667589229
[junit4:junit4] 2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=33), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=33)), sim=DefaultSimilarity, locale=ar_EG, timezone=Indian/Chagos
[junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=179264816,total=427163648
[junit4:junit4] 2> NOTE: All tests run in this JVM: [DirectUpdateHandlerOptimizeTest, FieldMutatingUpdateProcessorTest, TestSolrIndexConfig, CSVRequestHandlerTest, ChaosMonkeyNothingIsSafeTest, UUIDFieldTest, TestSearchPerf, TestPseudoReturnFields, SliceStateUpdateTest, TestExtendedDismaxParser, ZkControllerTest, TestManagedSchemaFieldResource, TestFieldTypeCollectionResource, RecoveryZkTest, TestPerFieldSimilarity, SolrInfoMBeanTest, SyncSliceTest, PrimUtilsTest, SuggesterTest, TestRandomDVFaceting, ScriptEngineTest, UpdateRequestProcessorFactoryTest, TestSolrDeletionPolicy2, QueryElevationComponentTest, AliasIntegrationTest, SuggesterFSTTest, LukeRequestHandlerTest, LegacyHTMLStripCharFilterTest, ShardRoutingTest, SpatialFilterTest, ClusterStateUpdateTest, TestIBSimilarityFactory, MoreLikeThisHandlerTest, TestReversedWildcardFilterFactory, RegexBoostProcessorTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RegexBoostProcessorTest -Dtests.seed=4CA14552DE06E6C0 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_EG -Dtests.timezone=Indian/Chagos -Dtests.file.encoding=ISO8859-1
[junit4:junit4] ERROR 0.00s J0 | RegexBoostProcessorTest (suite) <<<
[junit4:junit4] > Throwable #1: junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 22,137,840 bytes (threshold is 10,485,760). Field reference sizes (counted individually):
[junit4:junit4] > - 22,751,288 bytes, protected static org.apache.solr.servlet.SolrRequestParsers org.apache.solr.update.processor.RegexBoostProcessorTest._parser
[junit4:junit4] > - 5,256 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessorFactory org.apache.solr.update.processor.RegexBoostProcessorTest.factory
[junit4:junit4] > - 4,568 bytes, private static org.apache.solr.update.processor.RegexpBoostProcessor org.apache.solr.update.processor.RegexBoostProcessorTest.reProcessor
[junit4:junit4] > - 856 bytes, protected static org.apache.solr.common.params.ModifiableSolrParams org.apache.solr.update.processor.RegexBoostProcessorTest.parameters
[junit4:junit4] > - 296 bytes, protected static java.lang.String org.apache.solr.SolrTestCaseJ4.testSolrHome
[junit4:junit4] > - 272 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules
[junit4:junit4] > - 136 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp
[junit4:junit4] > - 72 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([4CA14552DE06E6C0]:0)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:679)
[junit4:junit4] Completed on J0 in 1.81s, 4 tests, 1 failure <<< FAILURES!
[...truncated 681 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:383: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:363: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1243: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:887: There were test failures: 296 suites, 1234 tests, 1 suite-level error, 1 error, 27 ignored (8 assumptions)
Total time: 59 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still Failing
Posted by Robert Muir <rc...@gmail.com>.
On Wed, May 15, 2013 at 9:12 PM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> Something is fishy with this sudden onset of RegexBoostProcessorTest
> failures....
>
> - happening on diff hosts, diff OSes, diff JVMs
> - happening on trunk and 4x
> - only started happening today
> - SolrRequestParsers object is huge for no explicible reason...
I think it might be related to its logger actually.
When i run it before mark's commit on SOLR-4825, its logger is 50kb
when i run it now, its logger is megabytes.
So I think that change caused it
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still Failing
Posted by Mark Miller <ma...@gmail.com>.
On May 16, 2013, at 12:59 PM, Mark Miller <ma...@gmail.com> wrote:
> In any case, it seems likely we can improve the Layout - I can take a look.
Seems like we could get away with just using SolrCore#hashCode rather than keying by the SolrCore.
- Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still Failing
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
> "The estimation actually includes static fields"
Yeah, I probably misunderstood something along the way, apologies.
When you look at the code I linked to things are pretty simple:
1) collect values of static fields from the test suite class (and its
superclasses),
2) for non-null values, estimate the RAM used by those objects and
their referenced graph of objects.
I think your understanding was correct (? :)
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still
Failing
Posted by Chris Hostetter <ho...@fucit.org>.
: > ...surely if the SolrRequestParser *class* had a static var the Logger
: > (instead of an instance var) that wouldn't have counted against the amount
: > of memory being used by the SolrRequestParser ?
:
: No, it wouldn't. We collect static fields from the suite class and
: then traverse the object graph from these (but don't count static
: fields again).
Now i'm completley confused. i said...
"the Logger used by SolrRequestParsers is not static, so the mem estimator
would count that as ram used by the object itself."
(the object in question being the SolrRequestParsers instance)
Then you said...
"The estimation actually includes static fields"
...it appeared that you were refering to static fields in the classes of
hte objects ithe estimation was asked to look at (ie: the
SolrRequestParsers class), but now i get the sense that you were refering
to static fields in the test -- which was never in question, all of the
fields listed in teh failure message are obviously static fields in the
test class.
So i think my initial understanding was correct, and we are just
confusing eachother and talking in circles?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still Failing
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
> ...surely if the SolrRequestParser *class* had a static var the Logger
> (instead of an instance var) that wouldn't have counted against the amount
> of memory being used by the SolrRequestParser ?
No, it wouldn't. We collect static fields from the suite class and
then traverse the object graph from these (but don't count static
fields again).
https://github.com/carrotsearch/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/rules/StaticFieldsInvariantRule.java#L91
Perhaps I should add an option to dump the heaviest objects (not just
fields) from that total graph estimator.
D.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still
Failing
Posted by Chris Hostetter <ho...@fucit.org>.
: > is not static, so the mem estimator would count that as ram used by the
: > object itself.
:
: The estimation actually includes static fields -- if it didn't you
: could easily overflow memory simply by allocating static byte[]
But clearly there is a diff between statics in the tests, and statics in
classes of objects held onto by the test -- otherwise we should have
started seeing way more "clean up your statics" failures then this when
the Logger changes got committed.
this test seemed to be the touchstone because...
* the test maintained a static, un-nulled out, SolrRequestParser
* SolrRequestParser instance have non-static refrences to a Logger
...surely if the SolrRequestParser *class* had a static var the Logger
(instead of an instance var) that wouldn't have counted against the amount
of memory being used by the SolrRequestParser ?
i mean ... if the static vars in every class were counted against the
objects of that class, the RAM estimater would be virtually useless.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still Failing
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
> is not static, so the mem estimator would count that as ram used by the
> object itself.
The estimation actually includes static fields -- if it didn't you
could easily overflow memory simply by allocating static byte[]
buffers or something else. Object-level fields are garbage collected
because the reference is released after the test (unless something
else keeps it; there is no way to tell). Take a look at the LTC:
.around(new StaticFieldsInvariantRule(STATIC_LEAK_THRESHOLD, true) {
@Override
protected boolean accept(java.lang.reflect.Field field) {
// Don't count known classes that consume memory once.
if (STATIC_LEAK_IGNORED_TYPES.contains(field.getType().getName())) {
return false;
}
// Don't count references from ourselves, we're top-level.
if (field.getDeclaringClass() == LuceneTestCase.class) {
return false;
}
return super.accept(field);
}
})
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still
Failing
Posted by Chris Hostetter <ho...@fucit.org>.
: What's a little odd is that this should work exactly as it did before we
It just occured to me that one thing rmuir and i noticed in IRC never made
it into email or my jira comments: the Logger used by SolrRequestParsers
is not static, so the mem estimator would count that as ram used by the
object itself.
I opened SOLR-4833 to audit all non-static Logger usage to try and
mitigate this type of failure in the future -- but that's still just
addressing the visible symptom that alerted us to (what i consider) the
more alarming problem: how big the Logger can get.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still Failing
Posted by Mark Miller <ma...@gmail.com>.
What's a little odd is that this should work exactly as it did before we used log4j - I just pure copied the code from a Formatter to a Layout. I originally suspected that the Layout is holding onto stuff, but thought perhaps that had something to do with java util logging vs log4j, since this didn't seem to previously show up as an issue. Perhaps we have added enough tests to cross a threshold while this was not in use.
In any case, it seems likely we can improve the Layout - I can take a look.
- Mark
On May 16, 2013, at 12:44 PM, Robert Muir <rc...@gmail.com> wrote:
> My suspicion is that its that map key'ed on solrcore.
>
> in this case it would be holding references to SolrCores from previous
> tests. Because test order and other things are randomized, it would
> explain the huge size fluctuations.
>
> On Thu, May 16, 2013 at 12:35 PM, Chris Hostetter
> <ho...@fucit.org> wrote:
>>
>>
>> : (b) is unlikely. Dump the allocation tree and see what's the major
>> : contributor to the size.
>>
>> Right, see rmuir's response and the subsequent collection of aggregation
>> of info in my comment on SOLR-4825...
>>
>> https://issues.apache.org/jira/browse/SOLR-4825#comment-13659117
>>
>>
>> -Hoss
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still Failing
Posted by Robert Muir <rc...@gmail.com>.
My suspicion is that its that map key'ed on solrcore.
in this case it would be holding references to SolrCores from previous
tests. Because test order and other things are randomized, it would
explain the huge size fluctuations.
On Thu, May 16, 2013 at 12:35 PM, Chris Hostetter
<ho...@fucit.org> wrote:
>
>
> : (b) is unlikely. Dump the allocation tree and see what's the major
> : contributor to the size.
>
> Right, see rmuir's response and the subsequent collection of aggregation
> of info in my comment on SOLR-4825...
>
> https://issues.apache.org/jira/browse/SOLR-4825#comment-13659117
>
>
> -Hoss
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still
Failing
Posted by Chris Hostetter <ho...@fucit.org>.
: (b) is unlikely. Dump the allocation tree and see what's the major
: contributor to the size.
Right, see rmuir's response and the subsequent collection of aggregation
of info in my comment on SOLR-4825...
https://issues.apache.org/jira/browse/SOLR-4825#comment-13659117
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still Failing
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
Hi Hoss,
> ...why is SolrRequestParsers 51 MB ?
> Instances of this class shouldn't by fluctuating in size, it doesn't
> maintain any mutable state -- so WTF?
You can try to debug it if you add an afterclass hook and dump the
"size tree" using, for example:
https://github.com/dweiss/java-sizeof/blob/master/src/main/java/com/carrotsearch/sizeof/ObjectTree.java
Typically what is happening is that there's a static field that holds
references to loggers, these in turn hold references to threads, these
to thread locals etc. Depends on the JVM and settings, really. I
assumed a class should nullify its static references after it's done
so an @AfterClass hook should be probably appropriate. Otherwise we
would miss statically allocated stuff and this can contribute to the
overall memory use.
> Don't get me wrong: RegexBoostProcessorTest should probably have an
> @AfterClass to null this out -- but i'm concerned that either:
Exactly.
> a) something has changed allowing SolrRequestParsers instances to now
> grow w/o bound
>
> b) something has changed to break the size detection code in the test
> framework.
(b) is unlikely. Dump the allocation tree and see what's the major
contributor to the size.
D.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 1621 - Still
Failing
Posted by Chris Hostetter <ho...@fucit.org>.
Something is fishy with this sudden onset of RegexBoostProcessorTest
failures....
- happening on diff hosts, diff OSes, diff JVMs
- happening on trunk and 4x
- only started happening today
- SolrRequestParsers object is huge for no explicible reason...
: Stack Trace: junit.framework.AssertionFailedError: Clean up static
: fields (in @AfterClass?), your test seems to hang on to approximately
: 51,150,032 bytes (threshold is 10,485,760). Field reference sizes
: (counted individually):
...
: - 51,880,376 bytes, protected static
: org.apache.solr.servlet.SolrRequestParsers
: org.apache.solr.update.processor.RegexBoostProcessorTest._parser
...why is SolrRequestParsers 51 MB ?
In other similar failures today it was 11MB, 12MB, 25MB, etc...
Instances of this class shouldn't by fluctuating in size, it doesn't
maintain any mutable state -- so WTF?
Don't get me wrong: RegexBoostProcessorTest should probably have an
@AfterClass to null this out -- but i'm concerned that either:
a) something has changed allowing SolrRequestParsers instances to now
grow w/o bound
b) something has changed to break the size detection code in the test
framework.
...anybody have any idea what changed?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org