You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/13 18:17:44 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b92) - Build #
6118 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6118/
Java: 32bit/jdk1.8.0-ea-b92 -client -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.core.TestMergePolicyConfig.testTieredMergePolicyConfig
Error Message:
expected:<0.0> but was:<1.0>
Stack Trace:
java.lang.AssertionError: expected:<0.0> but was:<1.0>
at __randomizedtesting.SeedInfo.seed([1ACBAE599F534DDC:B833B3DDC13EBBB7]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:443)
at org.junit.Assert.assertEquals(Assert.java:512)
at org.apache.solr.core.TestMergePolicyConfig.testTieredMergePolicyConfig(TestMergePolicyConfig.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9991 lines...]
[junit4:junit4] Suite: org.apache.solr.core.TestMergePolicyConfig
[junit4:junit4] 2> 920996 T3017 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385
[junit4:junit4] 2> 920997 T3017 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4] 2> 920997 T3017 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4] 2> 920998 T3017 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 921044 T3017 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 921079 T3017 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-mergepolicy.xml
[junit4:junit4] 2> 921079 T3017 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 921083 T3017 oass.IndexSchema.readSchema [null] Schema name=test
[junit4:junit4] 2> 921579 T3017 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 921587 T3017 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 921590 T3017 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 921597 T3017 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 921601 T3017 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 921604 T3017 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 921605 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 921605 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 921606 T3017 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 921606 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 921606 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 921607 T3017 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 921607 T3017 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr
[junit4:junit4] 2> 921608 T3017 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/'
[junit4:junit4] 2> 921622 T3017 oasc.CoreContainer.<init> New CoreContainer 28662875
[junit4:junit4] 2> 921665 T3017 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 921666 T3017 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385/
[junit4:junit4] 2> 921666 T3017 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4] 2> 921666 T3017 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 921667 T3017 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385
[junit4:junit4] 2> 921667 T3017 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385/index/
[junit4:junit4] 2> 921667 T3017 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 921668 T3017 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385/index
[junit4:junit4] 2> 921669 T3017 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@18f67a5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1218058),segFN=segments_1,generation=1}
[junit4:junit4] 2> 921669 T3017 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 921670 T3017 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 921673 T3017 oass.SolrIndexSearcher.<init> Opening Searcher@8755f1 main
[junit4:junit4] 2> 921673 T3017 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 921674 T3017 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 921674 T3017 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 921674 T3018 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8755f1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 921676 T3017 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4] 2> 921679 T3017 oas.SolrTestCaseJ4.setUp ###Starting testTieredMergePolicyConfig
[junit4:junit4] 2> 921680 T3017 oas.SolrTestCaseJ4.tearDown ###Ending testTieredMergePolicyConfig
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestMergePolicyConfig -Dtests.method=testTieredMergePolicyConfig -Dtests.seed=1ACBAE599F534DDC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_BH -Dtests.timezone=America/Aruba -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 0.02s J1 | TestMergePolicyConfig.testTieredMergePolicyConfig <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<0.0> but was:<1.0>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([1ACBAE599F534DDC:B833B3DDC13EBBB7]:0)
[junit4:junit4] > at org.apache.solr.core.TestMergePolicyConfig.testTieredMergePolicyConfig(TestMergePolicyConfig.java:43)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 921693 T3017 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 921693 T3017 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=28662875
[junit4:junit4] 2> 921693 T3017 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@122c2e5
[junit4:junit4] 2> 921693 T3017 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 921694 T3017 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 921694 T3017 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 921694 T3017 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 921695 T3017 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 921695 T3017 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 921695 T3017 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385;done=false>>]
[junit4:junit4] 2> 921695 T3017 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385
[junit4:junit4] 2> 921696 T3017 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385/index;done=false>>]
[junit4:junit4] 2> 921696 T3017 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371140057385/index
[junit4:junit4] 2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ar_BH, timezone=America/Aruba
[junit4:junit4] 2> NOTE: Linux 3.2.0-45-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=1,free=34492664,total=179683328
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SpellPossibilityIteratorTest, BadIndexSchemaTest, TestFiltering, StandardRequestHandlerTest, TestDFRSimilarityFactory, EchoParamsTest, TestSuggestSpellingConverter, TestSolrJ, LeaderElectionIntegrationTest, CollectionsAPIDistributedZkTest, OutputWriterTest, TestSolr4Spatial, TestZkChroot, TimeZoneUtilsTest, TermVectorComponentDistributedTest, TestUtils, TestHashPartitioner, NoCacheHeaderTest, NotRequiredUniqueKeyTest, XmlUpdateRequestHandlerTest, TestWordDelimiterFilterFactory, TestDefaultSearchFieldResource, SuggesterTSTTest, DateMathParserTest, TestCloudManagedSchema, TestRTGBase, BasicDistributedZk2Test, TestSolrXMLSerializer, FullSolrCloudDistribCmdsTest, TestSolrQueryParserResource, TestComponentsName, TestSolrXml, CurrencyFieldXmlFileTest, JSONWriterTest, SolrCoreCheckLockOnStartupTest, SuggesterWFSTTest, SolrPluginUtilsTest, ShardRoutingCustomTest, SolrRequestParserTest, TestDefaultSimilarityFactory, SortByFunctionTest, TestOmitPositions, QueryResultKeyTest, TestCSVLoader, TestReloadAndDeleteDocs, TestQuerySenderNoQuery, TestDynamicFieldResource, TestCodecSupport, TestAtomicUpdateErrorCases, TestStressReorder, URLClassifyProcessorTest, RequiredFieldsTest, DebugComponentTest, TestLazyCores, PathHierarchyTokenizerFactoryTest, SolrCmdDistributorTest, CurrencyFieldOpenExchangeTest, HighlighterTest, TestIndexSearcher, HardAutoCommitTest, TestElisionMultitermQuery, TestPropInject, DistanceFunctionTest, OpenCloseCoreStressTest, TestCoreDiscovery, TestPostingsSolrHighlighter, ZkSolrClientTest, TestQueryTypes, ChaosMonkeySafeLeaderTest, SyncSliceTest, OverseerTest, BasicZkTest, LeaderElectionTest, TestRandomDVFaceting, AutoCommitTest, TestFaceting, TestJoin, TestGroupingSearch, SimplePostToolTest, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SpellCheckCollatorTest, SuggesterFSTTest, TestFoldingMultitermQuery, SuggesterTest, SpatialFilterTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, TestSurroundQueryParser, DisMaxRequestHandlerTest, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, TermsComponentTest, TestIndexingPerformance, MoreLikeThisHandlerTest, TestArbitraryIndexDir, IndexSchemaRuntimeFieldTest, TestJmxIntegration, UpdateRequestProcessorFactoryTest, QueryParsingTest, UniqFieldsUpdateProcessorFactoryTest, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, HighlighterConfigTest, IndexReaderFactoryTest, CopyFieldTest, SolrIndexConfigTest, TestMergePolicyConfig]
[junit4:junit4] Completed on J1 in 0.72s, 1 test, 1 failure <<< FAILURES!
[...truncated 180 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:385: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:365: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1240: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 41 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b92 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build # 6119 - Still Failing!
Posted by Mark Miller <ma...@gmail.com>.
I created https://issues.apache.org/jira/browse/SOLR-4926
- Mark
On Jun 13, 2013, at 1:48 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6119/
> Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseG1GC
>
> 2 tests failed.
> REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
>
> Error Message:
> shard3 is not consistent. Got 49 from http://127.0.0.1:44371/collection1lastClient and got 47 from http://127.0.0.1:39672/collection1
>
> Stack Trace:
> java.lang.AssertionError: shard3 is not consistent. Got 49 from http://127.0.0.1:44371/collection1lastClient and got 47 from http://127.0.0.1:39672/collection1
> at __randomizedtesting.SeedInfo.seed([42BD083CD5E13AD1:C35B8624A2BE5AED]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
> at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
> at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at java.lang.Thread.run(Thread.java:722)
>
>
> FAILED: org.apache.solr.core.TestMergePolicyConfig.testTieredMergePolicyConfig
>
> Error Message:
> expected:<0.0> but was:<1.0>
>
> Stack Trace:
> java.lang.AssertionError: expected:<0.0> but was:<1.0>
> at __randomizedtesting.SeedInfo.seed([42BD083CD5E13AD1:E04515B88B8CCCBA]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:443)
> at org.junit.Assert.assertEquals(Assert.java:512)
> at org.apache.solr.core.TestMergePolicyConfig.testTieredMergePolicyConfig(TestMergePolicyConfig.java:43)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at java.lang.Thread.run(Thread.java:722)
>
>
>
>
> Build Log:
> [...truncated 9326 lines...]
> [junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
> [junit4:junit4] 2> 63014 T277 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
> [junit4:junit4] 2> 63031 T277 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
> [junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371144516372
> [junit4:junit4] 2> 63035 T277 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4:junit4] 2> 63038 T278 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
> [junit4:junit4] 2> 63138 T277 oasc.ZkTestServer.run start zk server on port:38719
> [junit4:junit4] 2> 63193 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 63305 T284 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23a60d52 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 63306 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 63308 T277 oascc.SolrZkClient.makePath makePath: /solr
> [junit4:junit4] 2> 63337 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 63339 T286 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@381e5476 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 63339 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 63340 T277 oascc.SolrZkClient.makePath makePath: /collections/collection1
> [junit4:junit4] 2> 63344 T277 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
> [junit4:junit4] 2> 63347 T277 oascc.SolrZkClient.makePath makePath: /collections/control_collection
> [junit4:junit4] 2> 63350 T277 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
> [junit4:junit4] 2> 63354 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
> [junit4:junit4] 2> 63355 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
> [junit4:junit4] 2> 63361 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
> [junit4:junit4] 2> 63362 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
> [junit4:junit4] 2> 63370 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
> [junit4:junit4] 2> 63370 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
> [junit4:junit4] 2> 63382 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
> [junit4:junit4] 2> 63383 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
> [junit4:junit4] 2> 63386 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
> [junit4:junit4] 2> 63386 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
> [junit4:junit4] 2> 63390 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
> [junit4:junit4] 2> 63390 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
> [junit4:junit4] 2> 63393 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4:junit4] 2> 63394 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4:junit4] 2> 63397 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
> [junit4:junit4] 2> 63397 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
> [junit4:junit4] 2> 63401 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
> [junit4:junit4] 2> 63401 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
> [junit4:junit4] 2> 63487 T277 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 63490 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50535
> [junit4:junit4] 2> 63491 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 63491 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 63491 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746
> [junit4:junit4] 2> 63492 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/solr.xml
> [junit4:junit4] 2> 63492 T277 oasc.CoreContainer.<init> New CoreContainer 899812787
> [junit4:junit4] 2> 63492 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/'
> [junit4:junit4] 2> 63493 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/'
> [junit4:junit4] 2> 63546 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 63546 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 63547 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 63547 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 63547 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 63548 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 63548 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 63549 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 63549 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 63549 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 63555 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 63555 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
> [junit4:junit4] 2> 63562 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 63563 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 63565 T297 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c6e6341 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 63566 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 63571 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 63580 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 63583 T299 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f729832 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 63583 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 63591 T277 oascc.SolrZkClient.makePath makePath: /overseer/queue
> [junit4:junit4] 2> 63602 T277 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
> [junit4:junit4] 2> 63612 T277 oascc.SolrZkClient.makePath makePath: /live_nodes
> [junit4:junit4] 2> 63616 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50535_
> [junit4:junit4] 2> 63620 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50535_
> [junit4:junit4] 2> 63627 T277 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
> [junit4:junit4] 2> 63641 T277 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
> [junit4:junit4] 2> 63644 T277 oasc.Overseer.start Overseer (id=89859327031181315-127.0.0.1:50535_-n_0000000000) starting
> [junit4:junit4] 2> 63655 T277 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
> [junit4:junit4] 2> 63666 T301 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
> [junit4:junit4] 2> 63667 T277 oascc.SolrZkClient.makePath makePath: /clusterstate.json
> [junit4:junit4] 2> 63670 T277 oascc.SolrZkClient.makePath makePath: /aliases.json
> [junit4:junit4] 2> 63673 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 63682 T300 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
> [junit4:junit4] 2> 63688 T302 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 63689 T302 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 65188 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 65190 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":null,
> [junit4:junit4] 2> "numShards":"1",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"control_collection",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:50535_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:50535"}
> [junit4:junit4] 2> 65190 T300 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
> [junit4:junit4] 2> 65193 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 65202 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4:junit4] 2> 65692 T302 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1
> [junit4:junit4] 2> 65692 T302 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
> [junit4:junit4] 2> 65693 T302 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 65693 T302 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
> [junit4:junit4] 2> 65695 T302 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1/'
> [junit4:junit4] 2> 65695 T302 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1/lib/README' to classloader
> [junit4:junit4] 2> 65696 T302 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 65741 T302 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
> [junit4:junit4] 2> 65791 T302 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 65793 T302 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 65798 T302 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 66166 T302 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 66173 T302 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 66176 T302 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 66194 T302 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 66198 T302 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 66202 T302 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 66203 T302 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 66203 T302 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 66203 T302 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 66205 T302 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 66205 T302 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 66205 T302 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 66212 T302 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/
> [junit4:junit4] 2> 66212 T302 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
> [junit4:junit4] 2> 66213 T302 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 66213 T302 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data
> [junit4:junit4] 2> 66214 T302 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/index/
> [junit4:junit4] 2> 66214 T302 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 66214 T302 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/index
> [junit4:junit4] 2> 66217 T302 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 66217 T302 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 66230 T302 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 66230 T302 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 66230 T302 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 66231 T302 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 66231 T302 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 66232 T302 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 66232 T302 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 66232 T302 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 66233 T302 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 66235 T302 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 66238 T302 oass.SolrIndexSearcher.<init> Opening Searcher@9fdecae main
> [junit4:junit4] 2> 66239 T302 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 66239 T302 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 66242 T303 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9fdecae main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 66244 T302 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 66244 T302 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50535 collection:control_collection shard:shard1
> [junit4:junit4] 2> 66247 T302 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
> [junit4:junit4] 2> 66253 T302 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
> [junit4:junit4] 2> 66267 T302 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 66279 T302 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 66279 T302 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 66281 T302 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50535/collection1/
> [junit4:junit4] 2> 66281 T302 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 66281 T302 oasc.SyncStrategy.syncToMe http://127.0.0.1:50535/collection1/ has no replicas
> [junit4:junit4] 2> 66281 T302 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50535/collection1/
> [junit4:junit4] 2> 66281 T302 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
> [junit4:junit4] 2> 66708 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 66816 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4:junit4] 2> 66842 T302 oasc.ZkController.register We are http://127.0.0.1:50535/collection1/ and leader is http://127.0.0.1:50535/collection1/
> [junit4:junit4] 2> 66843 T302 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50535
> [junit4:junit4] 2> 66843 T302 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 66843 T302 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 66843 T302 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 66846 T302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 66847 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 66848 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 66848 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 66854 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:
> [junit4:junit4] 2> 66857 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 66859 T306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38eb1748 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 66860 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 66861 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 66865 T277 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
> [junit4:junit4] 2> 66967 T277 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 66969 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59320
> [junit4:junit4] 2> 66970 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 66971 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 66971 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205
> [junit4:junit4] 2> 66971 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/solr.xml
> [junit4:junit4] 2> 66971 T277 oasc.CoreContainer.<init> New CoreContainer 106753595
> [junit4:junit4] 2> 66972 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/'
> [junit4:junit4] 2> 66972 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/'
> [junit4:junit4] 2> 67015 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 67016 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 67016 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 67017 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 67017 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 67017 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 67017 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 67018 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 67018 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 67018 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 67022 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 67023 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
> [junit4:junit4] 2> 67023 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 67024 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 67025 T317 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d9e4df7 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 67026 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 67029 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 67033 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 67035 T319 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57b356d4 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 67035 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 67040 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 68044 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59320_
> [junit4:junit4] 2> 68046 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59320_
> [junit4:junit4] 2> 68050 T299 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4] 2> 68050 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 68050 T306 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4] 2> 68051 T319 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4] 2> 68059 T320 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 68059 T320 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 68323 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 68325 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"127.0.0.1:50535__collection1",
> [junit4:junit4] 2> "numShards":"1",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"control_collection",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:50535_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:50535"}
> [junit4:junit4] 2> 68330 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":null,
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:59320_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:59320"}
> [junit4:junit4] 2> 68330 T300 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
> [junit4:junit4] 2> 68331 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 68367 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 68367 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 68367 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 69061 T320 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1
> [junit4:junit4] 2> 69061 T320 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 69062 T320 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 69062 T320 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 69064 T320 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1/'
> [junit4:junit4] 2> 69064 T320 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1/lib/README' to classloader
> [junit4:junit4] 2> 69065 T320 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 69091 T320 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
> [junit4:junit4] 2> 69156 T320 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 69158 T320 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 69162 T320 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 69507 T320 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 69514 T320 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 69516 T320 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 69536 T320 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 69541 T320 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 69546 T320 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 69547 T320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 69548 T320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 69548 T320 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 69550 T320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 69550 T320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 69550 T320 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 69551 T320 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/
> [junit4:junit4] 2> 69551 T320 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
> [junit4:junit4] 2> 69552 T320 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 69552 T320 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1
> [junit4:junit4] 2> 69553 T320 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index/
> [junit4:junit4] 2> 69553 T320 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 69553 T320 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index
> [junit4:junit4] 2> 69578 T320 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 69578 T320 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 69582 T320 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 69582 T320 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 69583 T320 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 69584 T320 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 69585 T320 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 69585 T320 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 69585 T320 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 69586 T320 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 69586 T320 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 69589 T320 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 69592 T320 oass.SolrIndexSearcher.<init> Opening Searcher@babf41f main
> [junit4:junit4] 2> 69593 T320 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 69594 T320 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 69599 T321 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@babf41f main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 69601 T320 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 69602 T320 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59320 collection:collection1 shard:shard1
> [junit4:junit4] 2> 69603 T320 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
> [junit4:junit4] 2> 69610 T320 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 69614 T320 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 69614 T320 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 69614 T320 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59320/collection1/
> [junit4:junit4] 2> 69614 T320 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 69614 T320 oasc.SyncStrategy.syncToMe http://127.0.0.1:59320/collection1/ has no replicas
> [junit4:junit4] 2> 69615 T320 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59320/collection1/
> [junit4:junit4] 2> 69615 T320 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
> [junit4:junit4] 2> 69877 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 69897 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 69897 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 69897 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 69923 T320 oasc.ZkController.register We are http://127.0.0.1:59320/collection1/ and leader is http://127.0.0.1:59320/collection1/
> [junit4:junit4] 2> 69923 T320 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59320
> [junit4:junit4] 2> 69923 T320 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 69924 T320 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 69924 T320 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 69926 T320 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 69928 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 69928 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 69929 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 69995 T277 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 69997 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45263
> [junit4:junit4] 2> 69999 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 69999 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 70000 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272
> [junit4:junit4] 2> 70000 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/solr.xml
> [junit4:junit4] 2> 70000 T277 oasc.CoreContainer.<init> New CoreContainer 768004907
> [junit4:junit4] 2> 70001 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/'
> [junit4:junit4] 2> 70001 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/'
> [junit4:junit4] 2> 70048 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 70048 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 70048 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 70049 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 70049 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 70049 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 70050 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 70050 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 70050 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 70051 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 70054 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 70055 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
> [junit4:junit4] 2> 70055 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 70056 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 70058 T333 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69b60e5d name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 70058 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 70061 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 70066 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 70068 T335 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71f2d728 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 70068 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 70073 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 71078 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45263_
> [junit4:junit4] 2> 71079 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45263_
> [junit4:junit4] 2> 71083 T306 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 71083 T299 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 71083 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 71083 T319 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 71083 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 71083 T335 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 71083 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 71089 T336 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 71090 T336 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 71409 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 71411 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"127.0.0.1:59320__collection1",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:59320_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:59320"}
> [junit4:junit4] 2> 71415 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":null,
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:45263_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:45263"}
> [junit4:junit4] 2> 71416 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 71416 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
> [junit4:junit4] 2> 71447 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 71447 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 71447 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 71447 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 72091 T336 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1
> [junit4:junit4] 2> 72091 T336 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 72092 T336 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 72092 T336 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 72094 T336 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1/'
> [junit4:junit4] 2> 72094 T336 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1/lib/README' to classloader
> [junit4:junit4] 2> 72095 T336 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 72122 T336 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
> [junit4:junit4] 2> 72156 T336 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 72157 T336 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 72162 T336 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 72514 T336 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 72521 T336 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 72523 T336 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 72539 T336 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 72543 T336 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 72546 T336 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 72548 T336 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 72548 T336 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 72548 T336 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 72550 T336 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 72550 T336 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 72550 T336 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 72550 T336 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/
> [junit4:junit4] 2> 72550 T336 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
> [junit4:junit4] 2> 72551 T336 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 72551 T336 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2
> [junit4:junit4] 2> 72551 T336 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index/
> [junit4:junit4] 2> 72552 T336 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 72552 T336 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index
> [junit4:junit4] 2> 72598 T336 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 72598 T336 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 72601 T336 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 72601 T336 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 72601 T336 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 72602 T336 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 72602 T336 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 72602 T336 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 72603 T336 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 72603 T336 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 72604 T336 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 72606 T336 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 72609 T336 oass.SolrIndexSearcher.<init> Opening Searcher@2f828d24 main
> [junit4:junit4] 2> 72610 T336 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 72610 T336 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 72614 T337 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2f828d24 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 72616 T336 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 72616 T336 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45263 collection:collection1 shard:shard2
> [junit4:junit4] 2> 72617 T336 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
> [junit4:junit4] 2> 72624 T336 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 72627 T336 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 72627 T336 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 72627 T336 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45263/collection1/
> [junit4:junit4] 2> 72628 T336 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 72628 T336 oasc.SyncStrategy.syncToMe http://127.0.0.1:45263/collection1/ has no replicas
> [junit4:junit4] 2> 72628 T336 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45263/collection1/
> [junit4:junit4] 2> 72628 T336 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
> [junit4:junit4] 2> 72956 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 72965 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 72966 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 72965 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 72966 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 72986 T336 oasc.ZkController.register We are http://127.0.0.1:45263/collection1/ and leader is http://127.0.0.1:45263/collection1/
> [junit4:junit4] 2> 72986 T336 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45263
> [junit4:junit4] 2> 72986 T336 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 72986 T336 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 72986 T336 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 72989 T336 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 72990 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 72991 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 72991 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 73067 T277 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 73070 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52190
> [junit4:junit4] 2> 73070 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 73070 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 73071 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336
> [junit4:junit4] 2> 73071 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/solr.xml
> [junit4:junit4] 2> 73071 T277 oasc.CoreContainer.<init> New CoreContainer 985733305
> [junit4:junit4] 2> 73072 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/'
> [junit4:junit4] 2> 73072 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/'
> [junit4:junit4] 2> 73118 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 73119 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 73119 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 73119 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 73119 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 73120 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 73120 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 73120 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 73121 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 73121 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 73125 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 73126 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
> [junit4:junit4] 2> 73126 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 73127 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 73128 T349 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@534b79e name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 73129 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 73130 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 73134 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 73136 T351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35e47b48 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 73136 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 73140 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 74144 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52190_
> [junit4:junit4] 2> 74146 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52190_
> [junit4:junit4] 2> 74150 T319 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 74150 T335 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 74150 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 74150 T306 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 74150 T351 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 74151 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 74151 T299 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 74150 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 74151 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 74157 T352 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 74157 T352 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 74473 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 74475 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"127.0.0.1:45263__collection1",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:45263_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:45263"}
> [junit4:junit4] 2> 74479 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":null,
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:52190_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:52190"}
> [junit4:junit4] 2> 74479 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 74479 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
> [junit4:junit4] 2> 74484 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 74484 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 74485 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 74485 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 74485 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 75158 T352 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1
> [junit4:junit4] 2> 75159 T352 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 75160 T352 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 75160 T352 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 75162 T352 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1/'
> [junit4:junit4] 2> 75162 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1/lib/README' to classloader
> [junit4:junit4] 2> 75163 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 75195 T352 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
> [junit4:junit4] 2> 75233 T352 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 75235 T352 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 75242 T352 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 75613 T352 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 75619 T352 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 75622 T352 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 75642 T352 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 75646 T352 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 75649 T352 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 75650 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 75651 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 75651 T352 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 75652 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 75652 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 75653 T352 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 75653 T352 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/
> [junit4:junit4] 2> 75653 T352 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
> [junit4:junit4] 2> 75654 T352 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 75654 T352 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3
> [junit4:junit4] 2> 75654 T352 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/index/
> [junit4:junit4] 2> 75654 T352 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 75655 T352 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/index
> [junit4:junit4] 2> 75658 T352 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 75659 T352 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 75661 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 75661 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 75662 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 75662 T352 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 75663 T352 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 75663 T352 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 75663 T352 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 75664 T352 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 75664 T352 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 75666 T352 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 75669 T352 oass.SolrIndexSearcher.<init> Opening Searcher@aa21710 main
> [junit4:junit4] 2> 75670 T352 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 75670 T352 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 75676 T353 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@aa21710 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 75678 T352 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 75678 T352 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52190 collection:collection1 shard:shard3
> [junit4:junit4] 2> 75679 T352 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
> [junit4:junit4] 2> 75686 T352 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 75689 T352 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 75689 T352 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 75689 T352 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52190/collection1/
> [junit4:junit4] 2> 75689 T352 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 75690 T352 oasc.SyncStrategy.syncToMe http://127.0.0.1:52190/collection1/ has no replicas
> [junit4:junit4] 2> 75690 T352 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52190/collection1/
> [junit4:junit4] 2> 75690 T352 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
> [junit4:junit4] 2> 75996 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 76005 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 76005 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 76005 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 76005 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 76005 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 76047 T352 oasc.ZkController.register We are http://127.0.0.1:52190/collection1/ and leader is http://127.0.0.1:52190/collection1/
> [junit4:junit4] 2> 76047 T352 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52190
> [junit4:junit4] 2> 76047 T352 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 76047 T352 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 76048 T352 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 76050 T352 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 76051 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 76051 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 76051 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 76118 T277 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 76120 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44393
> [junit4:junit4] 2> 76121 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 76121 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 76121 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395
> [junit4:junit4] 2> 76122 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/solr.xml
> [junit4:junit4] 2> 76122 T277 oasc.CoreContainer.<init> New CoreContainer 925394222
> [junit4:junit4] 2> 76122 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/'
> [junit4:junit4] 2> 76122 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/'
> [junit4:junit4] 2> 76164 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 76164 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 76164 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 76165 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 76165 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 76165 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 76166 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 76166 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 76166 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 76167 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 76170 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 76171 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
> [junit4:junit4] 2> 76171 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 76172 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 76174 T365 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@145c1572 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 76174 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 76176 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 76180 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 76181 T367 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c066577 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 76181 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 76185 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 77190 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44393_
> [junit4:junit4] 2> 77191 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44393_
> [junit4:junit4] 2> 77193 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 77195 T299 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 77195 T335 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 77195 T367 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 77195 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 77195 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 77197 T319 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 77197 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 77198 T306 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 77198 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 77200 T351 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 77206 T368 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 77206 T368 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 77514 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 77515 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"127.0.0.1:52190__collection1",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard3",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:52190_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:52190"}
> [junit4:junit4] 2> 77521 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":null,
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:44393_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:44393"}
> [junit4:junit4] 2> 77521 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 77521 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 77527 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 77527 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 77527 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 77527 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 77527 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 77528 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 78207 T368 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1
> [junit4:junit4] 2> 78208 T368 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 78209 T368 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 78209 T368 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 78211 T368 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1/'
> [junit4:junit4] 2> 78211 T368 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1/lib/README' to classloader
> [junit4:junit4] 2> 78212 T368 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 78248 T368 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
> [junit4:junit4] 2> 78285 T368 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 78287 T368 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 78291 T368 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 78608 T368 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 78615 T368 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 78617 T368 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 78638 T368 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 78643 T368 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 78648 T368 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 78650 T368 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 78650 T368 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 78651 T368 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 78652 T368 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 78652 T368 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 78653 T368 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 78653 T368 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/
> [junit4:junit4] 2> 78653 T368 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
> [junit4:junit4] 2> 78654 T368 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 78654 T368 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4
> [junit4:junit4] 2> 78655 T368 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index/
> [junit4:junit4] 2> 78655 T368 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 78656 T368 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index
> [junit4:junit4] 2> 78659 T368 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 78659 T368 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 78661 T368 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 78661 T368 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 78662 T368 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 78663 T368 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 78663 T368 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 78663 T368 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 78664 T368 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 78664 T368 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 78665 T368 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 78668 T368 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 78673 T368 oass.SolrIndexSearcher.<init> Opening Searcher@78e556dd main
> [junit4:junit4] 2> 78673 T368 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 78674 T368 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 78678 T369 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78e556dd main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 78680 T368 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 78680 T368 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44393 collection:collection1 shard:shard1
> [junit4:junit4] 2> 78684 T368 oasc.ZkController.register We are http://127.0.0.1:44393/collection1/ and leader is http://127.0.0.1:59320/collection1/
> [junit4:junit4] 2> 78684 T368 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44393
> [junit4:junit4] 2> 78684 T368 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 78684 T368 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C57 name=collection1 org.apache.solr.core.SolrCore@d189531 url=http://127.0.0.1:44393/collection1 node=127.0.0.1:44393_ C57_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:44393_, base_url=http://127.0.0.1:44393}
> [junit4:junit4] 2> 78687 T370 C57 P44393 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 78687 T368 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 78688 T370 C57 P44393 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 78689 T370 C57 P44393 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 78689 T370 C57 P44393 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 78689 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 78690 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 78690 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 78693 T370 C57 P44393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 78707 T312 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:44393__collection1, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 78768 T277 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 78771 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45641
> [junit4:junit4] 2> 78771 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 78772 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 78772 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035
> [junit4:junit4] 2> 78772 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/solr.xml
> [junit4:junit4] 2> 78773 T277 oasc.CoreContainer.<init> New CoreContainer 1946744004
> [junit4:junit4] 2> 78773 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/'
> [junit4:junit4] 2> 78773 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/'
> [junit4:junit4] 2> 78815 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 78816 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 78816 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 78816 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 78816 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 78817 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 78817 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 78817 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 78818 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 78818 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 78822 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 78822 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
> [junit4:junit4] 2> 78823 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 78824 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 78825 T382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b69c717 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 78825 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 78835 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 78840 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 78841 T384 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2612fc39 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 78841 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 78848 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 79035 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 79036 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"127.0.0.1:44393__collection1",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:44393_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:44393"}
> [junit4:junit4] 2> 79040 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79041 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79041 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79041 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79040 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79041 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79041 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79708 T312 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:44393__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4] 2> 79708 T312 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:44393__collection1&state=recovering&nodeName=127.0.0.1:44393_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
> [junit4:junit4] 2> 79852 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45641_
> [junit4:junit4] 2> 79853 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45641_
> [junit4:junit4] 2> 79855 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79856 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79856 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 79856 T299 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 79856 T306 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 79856 T335 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 79857 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 79856 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 79857 T319 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 79856 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 79857 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 79859 T367 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 79859 T351 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 79859 T384 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 79864 T385 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 79865 T385 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 80552 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 80553 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":null,
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:45641_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:45641"}
> [junit4:junit4] 2> 80553 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 80553 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
> [junit4:junit4] 2> 80558 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 80558 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 80558 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 80558 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 80558 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 80558 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 80558 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 80866 T385 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1
> [junit4:junit4] 2> 80866 T385 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 80867 T385 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 80868 T385 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 80869 T385 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1/'
> [junit4:junit4] 2> 80870 T385 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1/lib/README' to classloader
> [junit4:junit4] 2> 80870 T385 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 80894 T385 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
> [junit4:junit4] 2> 80922 T385 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 80924 T385 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 80928 T385 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 81248 T385 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 81254 T385 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 81256 T385 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 81272 T385 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 81276 T385 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 81279 T385 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 81280 T385 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 81281 T385 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 81281 T385 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 81282 T385 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 81282 T385 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 81283 T385 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 81283 T385 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/
> [junit4:junit4] 2> 81283 T385 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
> [junit4:junit4] 2> 81284 T385 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 81284 T385 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5
> [junit4:junit4] 2> 81284 T385 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index/
> [junit4:junit4] 2> 81285 T385 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 81285 T385 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index
> [junit4:junit4] 2> 81288 T385 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 81288 T385 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 81290 T385 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 81291 T385 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 81291 T385 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 81292 T385 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 81292 T385 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 81292 T385 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 81293 T385 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 81293 T385 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 81294 T385 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 81296 T385 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 81299 T385 oass.SolrIndexSearcher.<init> Opening Searcher@250b2251 main
> [junit4:junit4] 2> 81300 T385 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 81300 T385 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 81305 T386 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@250b2251 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 81307 T385 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 81307 T385 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45641 collection:collection1 shard:shard2
> [junit4:junit4] 2> 81311 T385 oasc.ZkController.register We are http://127.0.0.1:45641/collection1/ and leader is http://127.0.0.1:45263/collection1/
> [junit4:junit4] 2> 81311 T385 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45641
> [junit4:junit4] 2> 81311 T385 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 81311 T385 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C58 name=collection1 org.apache.solr.core.SolrCore@2014e6de url=http://127.0.0.1:45641/collection1 node=127.0.0.1:45641_ C58_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:45641_, base_url=http://127.0.0.1:45641}
> [junit4:junit4] 2> 81312 T387 C58 P45641 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 81312 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 81313 T387 C58 P45641 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 81313 T387 C58 P45641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 81313 T385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 81314 T387 C58 P45641 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 81315 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 81315 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 81316 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 81319 T330 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:45641__collection1, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 81396 T277 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 81399 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45982
> [junit4:junit4] 2> 81400 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 81400 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 81400 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659
> [junit4:junit4] 2> 81401 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/solr.xml
> [junit4:junit4] 2> 81401 T277 oasc.CoreContainer.<init> New CoreContainer 243591166
> [junit4:junit4] 2> 81402 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/'
> [junit4:junit4] 2> 81402 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/'
> [junit4:junit4] 2> 81449 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 81449 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 81450 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 81450 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 81450 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 81450 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 81451 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 81451 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 81451 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 81452 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 81457 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 81457 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
> [junit4:junit4] 2> 81458 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 81459 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 81461 T399 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51e1ca54 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 81462 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 81464 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 81481 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 81481 T401 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@186a5fac name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 81482 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 81486 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> ASYNC NEW_CORE C59 name=collection1 org.apache.solr.core.SolrCore@d189531 url=http://127.0.0.1:44393/collection1 node=127.0.0.1:44393_ C59_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:44393_, base_url=http://127.0.0.1:44393}
> [junit4:junit4] 2> 81709 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59320/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4] 2> 81714 T370 C59 P44393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
> [junit4:junit4] 2> 81721 T370 C59 P44393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44393 START replicas=[http://127.0.0.1:59320/collection1/] nUpdates=100
> [junit4:junit4] 2> 81722 T370 C59 P44393 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4] 2> 81722 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4] 2> 81722 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4] 2> 81722 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4] 2> 81723 T370 C59 P44393 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4] 2> 81723 T370 C59 P44393 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59320/collection1/. core=collection1
> [junit4:junit4] 2> 81723 T370 C59 P44393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> ASYNC NEW_CORE C60 name=collection1 org.apache.solr.core.SolrCore@7ff4ca39 url=http://127.0.0.1:59320/collection1 node=127.0.0.1:59320_ C60_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59320_, base_url=http://127.0.0.1:59320, leader=true}
> [junit4:junit4] 2> 81728 T313 C60 P59320 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
> [junit4:junit4] 2> 81731 T315 C60 P59320 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4] 2> 81733 T315 C60 P59320 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 81734 T315 C60 P59320 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 81746 T315 C60 P59320 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index,segFN=segments_2,generation=2}
> [junit4:junit4] 2> 81747 T315 C60 P59320 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
> [junit4:junit4] 2> 81748 T315 C60 P59320 oass.SolrIndexSearcher.<init> Opening Searcher@7f3220e3 realtime
> [junit4:junit4] 2> 81748 T315 C60 P59320 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4] 2> 81748 T315 C60 P59320 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
> [junit4:junit4] 2> 81773 T370 C59 P44393 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 81774 T370 C59 P44393 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
> [junit4:junit4] 2> 81782 T370 C59 P44393 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4] 2> 81785 T312 C60 P59320 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 81786 T312 C60 P59320 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
> [junit4:junit4] 2> 81787 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Master's generation: 2
> [junit4:junit4] 2> 81787 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
> [junit4:junit4] 2> 81787 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Starting replication process
> [junit4:junit4] 2> 81790 T313 C60 P59320 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
> [junit4:junit4] 2> 81790 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
> [junit4:junit4] 2> 81791 T370 C59 P44393 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131
> [junit4:junit4] 2> 81792 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131 lockFactory=org.apache.lucene.store.NativeFSLockFactory@77347fda fullCopy=false
> [junit4:junit4] 2> 81799 T314 C60 P59320 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=2
> [junit4:junit4] 2> 81804 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
> [junit4:junit4] 2> 81807 T370 C59 P44393 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
> [junit4:junit4] 2> 81808 T370 C59 P44393 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
> [junit4:junit4] 2> 81812 T370 C59 P44393 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index,segFN=segments_2,generation=2}
> [junit4:junit4] 2> 81813 T370 C59 P44393 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
> [junit4:junit4] 2> 81813 T370 C59 P44393 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
> [junit4:junit4] 2> 81813 T370 C59 P44393 oass.SolrIndexSearcher.<init> Opening Searcher@1511258c main
> [junit4:junit4] 2> 81815 T369 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1511258c main{StandardDirectoryReader(segments_2:1:nrt)}
> [junit4:junit4] 2> 81816 T370 C59 P44393 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131;done=true>>]
> [junit4:junit4] 2> 81816 T370 C59 P44393 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131
> [junit4:junit4] 2> 81816 T370 C59 P44393 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131
> [junit4:junit4] 2> 81820 T370 C59 P44393 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4] 2> 81820 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4] 2> 81821 T370 C59 P44393 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 81821 T370 C59 P44393 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 81823 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4] 2> 82065 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 82066 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"127.0.0.1:45641__collection1",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:45641_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:45641"}
> [junit4:junit4] 2> 82069 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"127.0.0.1:44393__collection1",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:44393_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:44393"}
> [junit4:junit4] 2> 82087 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82087 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82087 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82087 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82087 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82087 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82087 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82087 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82320 T330 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:45641__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4] 2> 82320 T330 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:45641__collection1&state=recovering&nodeName=127.0.0.1:45641_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
> [junit4:junit4] 2> 82490 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45982_
> [junit4:junit4] 2> 82491 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45982_
> [junit4:junit4] 2> 82494 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82494 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82495 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82495 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 82495 T335 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 82495 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 82496 T299 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 82496 T306 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 82496 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 82496 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 82496 T319 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 82497 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 82498 T384 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 82499 T401 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 82499 T367 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 82499 T351 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 82506 T404 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 82506 T404 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 83598 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 83599 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":null,
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:45982_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:45982"}
> [junit4:junit4] 2> 83599 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 83599 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
> [junit4:junit4] 2> 83605 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 83605 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 83605 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 83605 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 83605 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 83605 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 83605 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 83605 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> C58_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:45641_, base_url=http://127.0.0.1:45641}
> [junit4:junit4] 2> 84321 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45263/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4] 2> 84322 T387 C58 P45641 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45641 START replicas=[http://127.0.0.1:45263/collection1/] nUpdates=100
> [junit4:junit4] 2> 84322 T387 C58 P45641 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4] 2> 84323 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4] 2> 84323 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4] 2> 84323 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4] 2> 84324 T387 C58 P45641 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4] 2> 84324 T387 C58 P45641 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45263/collection1/. core=collection1
> [junit4:junit4] 2> 84324 T387 C58 P45641 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> ASYNC NEW_CORE C61 name=collection1 org.apache.solr.core.SolrCore@4fd686fc url=http://127.0.0.1:45263/collection1 node=127.0.0.1:45263_ C61_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:45263_, base_url=http://127.0.0.1:45263, leader=true}
> [junit4:junit4] 2> 84328 T330 C61 P45263 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
> [junit4:junit4] 2> 84330 T328 C61 P45263 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4] 2> 84333 T328 C61 P45263 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 84335 T328 C61 P45263 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 84346 T328 C61 P45263 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index,segFN=segments_2,generation=2}
> [junit4:junit4] 2> 84347 T328 C61 P45263 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
> [junit4:junit4] 2> 84348 T328 C61 P45263 oass.SolrIndexSearcher.<init> Opening Searcher@2e358e22 realtime
> [junit4:junit4] 2> 84348 T328 C61 P45263 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4] 2> 84348 T328 C61 P45263 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
> [junit4:junit4] 2> 84349 T387 C58 P45641 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 84350 T387 C58 P45641 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4] 2> 84352 T331 C61 P45263 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 84353 T331 C61 P45263 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
> [junit4:junit4] 2> 84353 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Master's generation: 2
> [junit4:junit4] 2> 84353 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
> [junit4:junit4] 2> 84354 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Starting replication process
> [junit4:junit4] 2> 84356 T330 C61 P45263 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
> [junit4:junit4] 2> 84357 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
> [junit4:junit4] 2> 84357 T387 C58 P45641 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697
> [junit4:junit4] 2> 84358 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697 lockFactory=org.apache.lucene.store.NativeFSLockFactory@562cf72d fullCopy=false
> [junit4:junit4] 2> 84361 T328 C61 P45263 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
> [junit4:junit4] 2> 84362 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
> [junit4:junit4] 2> 84367 T387 C58 P45641 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
> [junit4:junit4] 2> 84367 T387 C58 P45641 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
> [junit4:junit4] 2> 84370 T387 C58 P45641 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index,segFN=segments_2,generation=2}
> [junit4:junit4] 2> 84370 T387 C58 P45641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
> [junit4:junit4] 2> 84371 T387 C58 P45641 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
> [junit4:junit4] 2> 84371 T387 C58 P45641 oass.SolrIndexSearcher.<init> Opening Searcher@33c346d7 main
> [junit4:junit4] 2> 84372 T386 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33c346d7 main{StandardDirectoryReader(segments_2:1:nrt)}
> [junit4:junit4] 2> 84373 T387 C58 P45641 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697;done=true>>]
> [junit4:junit4] 2> 84373 T387 C58 P45641 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697
> [junit4:junit4] 2> 84373 T387 C58 P45641 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697
> [junit4:junit4] 2> 84373 T387 C58 P45641 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4] 2> 84373 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4] 2> 84374 T387 C58 P45641 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 84374 T387 C58 P45641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 84376 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4] 2> 84508 T404 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1
> [junit4:junit4] 2> 84508 T404 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 84509 T404 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 84509 T404 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 84510 T404 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1/'
> [junit4:junit4] 2> 84511 T404 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1/lib/README' to classloader
> [junit4:junit4] 2> 84511 T404 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 84539 T404 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
> [junit4:junit4] 2> 84568 T404 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 84570 T404 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 84574 T404 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 84915 T404 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 84922 T404 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 84924 T404 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 84942 T404 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 84947 T404 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 84952 T404 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 84954 T404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 84954 T404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 84955 T404 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 84956 T404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 84956 T404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 84957 T404 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 84957 T404 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/
> [junit4:junit4] 2> 84957 T404 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
> [junit4:junit4] 2> 84958 T404 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 84959 T404 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6
> [junit4:junit4] 2> 84959 T404 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/index/
> [junit4:junit4] 2> 84959 T404 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 84960 T404 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/index
> [junit4:junit4] 2> 84978 T404 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 84979 T404 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 84981 T404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 84981 T404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 84982 T404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 84982 T404 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 84983 T404 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 84983 T404 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 84986 T404 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 84986 T404 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 84987 T404 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 84989 T404 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 84992 T404 oass.SolrIndexSearcher.<init> Opening Searcher@7c14f9fd main
> [junit4:junit4] 2> 84992 T404 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 84993 T404 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 84997 T407 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c14f9fd main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 84999 T404 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 84999 T404 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45982 collection:collection1 shard:shard3
> [junit4:junit4] 2> 85003 T404 oasc.ZkController.register We are http://127.0.0.1:45982/collection1/ and leader is http://127.0.0.1:52190/collection1/
> [junit4:junit4] 2> 85003 T404 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45982
> [junit4:junit4] 2> 85003 T404 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 85003 T404 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C62 name=collection1 org.apache.solr.core.SolrCore@2ce806a3 url=http://127.0.0.1:45982/collection1 node=127.0.0.1:45982_ C62_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:45982_, base_url=http://127.0.0.1:45982}
> [junit4:junit4] 2> 85004 T408 C62 P45982 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 85004 T404 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 85004 T408 C62 P45982 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 85005 T408 C62 P45982 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 85005 T408 C62 P45982 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 85006 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 85006 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 85007 T408 C62 P45982 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 85007 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 85012 T347 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:45982__collection1, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 85099 T277 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 85102 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41375
> [junit4:junit4] 2> 85102 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 85102 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 85103 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352
> [junit4:junit4] 2> 85103 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/solr.xml
> [junit4:junit4] 2> 85103 T277 oasc.CoreContainer.<init> New CoreContainer 28452598
> [junit4:junit4] 2> 85104 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/'
> [junit4:junit4] 2> 85104 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/'
> [junit4:junit4] 2> 85112 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 85113 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"127.0.0.1:45641__collection1",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:45641_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:45641"}
> [junit4:junit4] 2> 85116 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"127.0.0.1:45982__collection1",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard3",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:45982_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:45982"}
> [junit4:junit4] 2> 85121 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 85121 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 85121 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 85122 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 85121 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 85121 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 85121 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 85122 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 85152 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 85153 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 85153 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 85153 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 85154 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 85154 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 85154 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 85155 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 85155 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 85155 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 85159 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 85160 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
> [junit4:junit4] 2> 85160 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 85161 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 85163 T420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46464845 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 85163 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 85164 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 85169 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 85170 T422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ab27676 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 85171 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 85175 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 86013 T347 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:45982__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4] 2> 86013 T347 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:45982__collection1&state=recovering&nodeName=127.0.0.1:45982_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
> [junit4:junit4] 2> 86179 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41375_
> [junit4:junit4] 2> 86181 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41375_
> [junit4:junit4] 2> 86183 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 86184 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 86184 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 86184 T306 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 86184 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 86185 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86185 T335 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 86185 T299 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 86185 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86185 T319 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 86186 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86185 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86186 T422 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 86188 T367 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 86188 T384 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 86187 T401 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 86188 T351 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 86193 T423 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 86193 T423 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 86630 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 86631 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":null,
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:41375_",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:41375"}
> [junit4:junit4] 2> 86631 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 86632 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 86637 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86637 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86637 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86637 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86637 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86637 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86637 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86637 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 86637 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 87195 T423 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1
> [junit4:junit4] 2> 87195 T423 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 87196 T423 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 87196 T423 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 87198 T423 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1/'
> [junit4:junit4] 2> 87198 T423 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1/lib/README' to classloader
> [junit4:junit4] 2> 87199 T423 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 87228 T423 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
> [junit4:junit4] 2> 87258 T423 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 87260 T423 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 87264 T423 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 87620 T423 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 87627 T423 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 87629 T423 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 87648 T423 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 87651 T423 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 87655 T423 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 87657 T423 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 87657 T423 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 87657 T423 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 87658 T423 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 87659 T423 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 87659 T423 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 87659 T423 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/
> [junit4:junit4] 2> 87659 T423 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
> [junit4:junit4] 2> 87660 T423 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 87660 T423 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7
> [junit4:junit4] 2> 87661 T423 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/index/
> [junit4:junit4] 2> 87661 T423 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 87661 T423 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/index
> [junit4:junit4] 2> 87664 T423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 87664 T423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 87666 T423 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 87667 T423 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 87667 T423 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 87668 T423 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 87668 T423 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 87668 T423 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 87669 T423 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 87669 T423 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 87670 T423 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 87672 T423 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 87675 T423 oass.SolrIndexSearcher.<init> Opening Searcher@3856fa9e main
> [junit4:junit4] 2> 87675 T423 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 87676 T423 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 87682 T424 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3856fa9e main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 87685 T423 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 87685 T423 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41375 collection:collectio
>
> [...truncated too long message...]
>
> OrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 366507 T1266 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 366507 T1266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 366507 T1266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 366507 T1266 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 366508 T1266 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr
> [junit4:junit4] 2> 366508 T1266 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/'
> [junit4:junit4] 2> 366516 T1266 oasc.CoreContainer.<init> New CoreContainer 1238919290
> [junit4:junit4] 2> 366541 T1266 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
> [junit4:junit4] 2> 366541 T1266 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/
> [junit4:junit4] 2> 366541 T1266 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
> [junit4:junit4] 2> 366542 T1266 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
> [junit4:junit4] 2> 366542 T1266 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357
> [junit4:junit4] 2> 366542 T1266 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index/
> [junit4:junit4] 2> 366543 T1266 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 366543 T1266 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index
> [junit4:junit4] 2> 366544 T1266 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5510ce87 lockFactory=org.apache.lucene.store.NativeFSLockFactory@68c9703a),segFN=segments_1,generation=1}
> [junit4:junit4] 2> 366545 T1266 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 366546 T1266 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 366598 T1266 oass.SolrIndexSearcher.<init> Opening Searcher@67713565 main
> [junit4:junit4] 2> 366598 T1266 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 366599 T1266 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 366599 T1266 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 366599 T1267 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67713565 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 366601 T1266 oas.SolrTestCaseJ4.initCore ####initCore end
> [junit4:junit4] 2> 366605 T1266 oas.SolrTestCaseJ4.setUp ###Starting testTieredMergePolicyConfig
> [junit4:junit4] 2> 366605 T1266 oas.SolrTestCaseJ4.tearDown ###Ending testTieredMergePolicyConfig
> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestMergePolicyConfig -Dtests.method=testTieredMergePolicyConfig -Dtests.seed=42BD083CD5E13AD1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=no -Dtests.timezone=America/Noronha -Dtests.file.encoding=ISO-8859-1
> [junit4:junit4] FAILURE 0.01s J1 | TestMergePolicyConfig.testTieredMergePolicyConfig <<<
> [junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<0.0> but was:<1.0>
> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([42BD083CD5E13AD1:E04515B88B8CCCBA]:0)
> [junit4:junit4] > at org.apache.solr.core.TestMergePolicyConfig.testTieredMergePolicyConfig(TestMergePolicyConfig.java:43)
> [junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4] 2> 366610 T1266 oas.SolrTestCaseJ4.deleteCore ###deleteCore
> [junit4:junit4] 2> 366610 T1266 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1238919290
> [junit4:junit4] 2> 366610 T1266 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@6fc0e4e3
> [junit4:junit4] 2> 366610 T1266 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
> [junit4:junit4] 2> 366611 T1266 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
> [junit4:junit4] 2> 366611 T1266 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
> [junit4:junit4] 2> 366611 T1266 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
> [junit4:junit4] 2> 366611 T1266 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
> [junit4:junit4] 2> 366612 T1266 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
> [junit4:junit4] 2> 366612 T1266 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357;done=false>>]
> [junit4:junit4] 2> 366612 T1266 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357
> [junit4:junit4] 2> 366612 T1266 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index;done=false>>]
> [junit4:junit4] 2> 366613 T1266 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index
> [junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {}, docValues:{}, sim=DefaultSimilarity, locale=no, timezone=America/Noronha
> [junit4:junit4] 2> NOTE: Linux 3.2.0-45-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=1,free=373635368,total=521142272
> [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestBM25SimilarityFactory, HighlighterConfigTest, TestDistributedSearch, TestSchemaNameResource, DefaultValueUpdateProcessorTest, SolrIndexConfigTest, TestStressLucene, TestSolrXMLSerializer, TestOmitPositions, JSONWriterTest, TestBinaryResponseWriter, TestLMJelinekMercerSimilarityFactory, TestFieldResource, ClusterStateTest, TestCopyFieldCollectionResource, TestRangeQuery, ChaosMonkeySafeLeaderTest, TestFieldTypeCollectionResource, TestSolrJ, NotRequiredUniqueKeyTest, DocumentAnalysisRequestHandlerTest, PathHierarchyTokenizerFactoryTest, IndexBasedSpellCheckerTest, SolrRequestParserTest, TestCSVResponseWriter, BasicDistributedZk2Test, PingRequestHandlerTest, MultiTermTest, SortByFunctionTest, TestDistributedGrouping, StandardRequestHandlerTest, OutputWriterTest, RequiredFieldsTest, TestIndexingPerformance, TestMultiCoreConfBootstrap, SuggesterWFSTTest, TestSolrQueryParserDefaultOperatorResource, CurrencyFieldXmlFileTest, DateMathParserTest, TestAtomicUpdateErrorCases, SolrCoreCheckLockOnStartupTest, TestDefaultSimilarityFactory, TestComponentsName, TestDynamicFieldCollectionResource, TestMergePolicyConfig]
> [junit4:junit4] Completed on J1 in 0.61s, 1 test, 1 failure <<< FAILURES!
>
> [...truncated 565 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:385: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:365: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1240: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 2 failures, 13 ignored (7 assumptions)
>
> Total time: 39 minutes 18 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseG1GC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
6119 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6119/
Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseG1GC
2 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard3 is not consistent. Got 49 from http://127.0.0.1:44371/collection1lastClient and got 47 from http://127.0.0.1:39672/collection1
Stack Trace:
java.lang.AssertionError: shard3 is not consistent. Got 49 from http://127.0.0.1:44371/collection1lastClient and got 47 from http://127.0.0.1:39672/collection1
at __randomizedtesting.SeedInfo.seed([42BD083CD5E13AD1:C35B8624A2BE5AED]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
FAILED: org.apache.solr.core.TestMergePolicyConfig.testTieredMergePolicyConfig
Error Message:
expected:<0.0> but was:<1.0>
Stack Trace:
java.lang.AssertionError: expected:<0.0> but was:<1.0>
at __randomizedtesting.SeedInfo.seed([42BD083CD5E13AD1:E04515B88B8CCCBA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:443)
at org.junit.Assert.assertEquals(Assert.java:512)
at org.apache.solr.core.TestMergePolicyConfig.testTieredMergePolicyConfig(TestMergePolicyConfig.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9326 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 63014 T277 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 63031 T277 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371144516372
[junit4:junit4] 2> 63035 T277 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 63038 T278 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 63138 T277 oasc.ZkTestServer.run start zk server on port:38719
[junit4:junit4] 2> 63193 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 63305 T284 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23a60d52 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 63306 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 63308 T277 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 63337 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 63339 T286 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@381e5476 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 63339 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 63340 T277 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 63344 T277 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 63347 T277 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 63350 T277 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 63354 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 63355 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 63361 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 63362 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 63370 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 63370 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 63382 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 63383 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 63386 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 63386 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 63390 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 63390 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 63393 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 63394 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 63397 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 63397 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 63401 T277 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 63401 T277 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 63487 T277 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 63490 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50535
[junit4:junit4] 2> 63491 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 63491 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 63491 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746
[junit4:junit4] 2> 63492 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/solr.xml
[junit4:junit4] 2> 63492 T277 oasc.CoreContainer.<init> New CoreContainer 899812787
[junit4:junit4] 2> 63492 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/'
[junit4:junit4] 2> 63493 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/'
[junit4:junit4] 2> 63546 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 63546 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 63547 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 63547 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 63547 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 63548 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 63548 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 63549 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 63549 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 63549 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 63555 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 63555 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
[junit4:junit4] 2> 63562 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 63563 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 63565 T297 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c6e6341 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 63566 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 63571 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 63580 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 63583 T299 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f729832 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 63583 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 63591 T277 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 63602 T277 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 63612 T277 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 63616 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50535_
[junit4:junit4] 2> 63620 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50535_
[junit4:junit4] 2> 63627 T277 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 63641 T277 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 63644 T277 oasc.Overseer.start Overseer (id=89859327031181315-127.0.0.1:50535_-n_0000000000) starting
[junit4:junit4] 2> 63655 T277 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 63666 T301 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 63667 T277 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 63670 T277 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 63673 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 63682 T300 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 63688 T302 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 63689 T302 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 65188 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 65190 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50535_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50535"}
[junit4:junit4] 2> 65190 T300 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 65193 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 65202 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 65692 T302 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1
[junit4:junit4] 2> 65692 T302 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 65693 T302 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 65693 T302 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 65695 T302 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1/'
[junit4:junit4] 2> 65695 T302 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1/lib/README' to classloader
[junit4:junit4] 2> 65696 T302 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 65741 T302 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 65791 T302 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 65793 T302 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 65798 T302 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 66166 T302 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 66173 T302 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 66176 T302 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 66194 T302 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 66198 T302 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 66202 T302 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 66203 T302 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 66203 T302 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 66203 T302 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 66205 T302 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 66205 T302 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 66205 T302 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 66212 T302 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371144516746/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/
[junit4:junit4] 2> 66212 T302 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
[junit4:junit4] 2> 66213 T302 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 66213 T302 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data
[junit4:junit4] 2> 66214 T302 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/index/
[junit4:junit4] 2> 66214 T302 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 66214 T302 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/index
[junit4:junit4] 2> 66217 T302 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 66217 T302 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 66230 T302 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 66230 T302 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 66230 T302 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 66231 T302 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 66231 T302 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 66232 T302 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 66232 T302 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 66232 T302 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 66233 T302 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 66235 T302 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 66238 T302 oass.SolrIndexSearcher.<init> Opening Searcher@9fdecae main
[junit4:junit4] 2> 66239 T302 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 66239 T302 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 66242 T303 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9fdecae main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 66244 T302 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 66244 T302 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50535 collection:control_collection shard:shard1
[junit4:junit4] 2> 66247 T302 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 66253 T302 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 66267 T302 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 66279 T302 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 66279 T302 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 66281 T302 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50535/collection1/
[junit4:junit4] 2> 66281 T302 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 66281 T302 oasc.SyncStrategy.syncToMe http://127.0.0.1:50535/collection1/ has no replicas
[junit4:junit4] 2> 66281 T302 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50535/collection1/
[junit4:junit4] 2> 66281 T302 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 66708 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 66816 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 66842 T302 oasc.ZkController.register We are http://127.0.0.1:50535/collection1/ and leader is http://127.0.0.1:50535/collection1/
[junit4:junit4] 2> 66843 T302 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50535
[junit4:junit4] 2> 66843 T302 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 66843 T302 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 66843 T302 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 66846 T302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 66847 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 66848 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 66848 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 66854 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 66857 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 66859 T306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38eb1748 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 66860 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 66861 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 66865 T277 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 66967 T277 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 66969 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59320
[junit4:junit4] 2> 66970 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 66971 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 66971 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205
[junit4:junit4] 2> 66971 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/solr.xml
[junit4:junit4] 2> 66971 T277 oasc.CoreContainer.<init> New CoreContainer 106753595
[junit4:junit4] 2> 66972 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/'
[junit4:junit4] 2> 66972 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/'
[junit4:junit4] 2> 67015 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 67016 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 67016 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 67017 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 67017 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 67017 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 67017 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 67018 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 67018 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 67018 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 67022 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 67023 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
[junit4:junit4] 2> 67023 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 67024 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 67025 T317 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d9e4df7 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 67026 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 67029 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 67033 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 67035 T319 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57b356d4 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 67035 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 67040 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 68044 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59320_
[junit4:junit4] 2> 68046 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59320_
[junit4:junit4] 2> 68050 T299 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 68050 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 68050 T306 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 68051 T319 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 68059 T320 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 68059 T320 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 68323 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 68325 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50535__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50535_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50535"}
[junit4:junit4] 2> 68330 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59320_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59320"}
[junit4:junit4] 2> 68330 T300 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 68331 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 68367 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 68367 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 68367 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 69061 T320 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1
[junit4:junit4] 2> 69061 T320 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 69062 T320 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 69062 T320 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 69064 T320 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1/'
[junit4:junit4] 2> 69064 T320 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1/lib/README' to classloader
[junit4:junit4] 2> 69065 T320 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 69091 T320 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 69156 T320 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 69158 T320 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 69162 T320 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 69507 T320 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 69514 T320 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 69516 T320 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 69536 T320 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 69541 T320 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 69546 T320 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 69547 T320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 69548 T320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 69548 T320 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 69550 T320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 69550 T320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 69550 T320 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 69551 T320 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371144520205/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/
[junit4:junit4] 2> 69551 T320 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
[junit4:junit4] 2> 69552 T320 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 69552 T320 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1
[junit4:junit4] 2> 69553 T320 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index/
[junit4:junit4] 2> 69553 T320 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 69553 T320 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index
[junit4:junit4] 2> 69578 T320 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 69578 T320 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 69582 T320 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 69582 T320 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 69583 T320 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 69584 T320 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 69585 T320 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 69585 T320 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 69585 T320 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 69586 T320 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 69586 T320 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 69589 T320 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 69592 T320 oass.SolrIndexSearcher.<init> Opening Searcher@babf41f main
[junit4:junit4] 2> 69593 T320 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 69594 T320 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 69599 T321 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@babf41f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 69601 T320 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 69602 T320 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59320 collection:collection1 shard:shard1
[junit4:junit4] 2> 69603 T320 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 69610 T320 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 69614 T320 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 69614 T320 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 69614 T320 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59320/collection1/
[junit4:junit4] 2> 69614 T320 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 69614 T320 oasc.SyncStrategy.syncToMe http://127.0.0.1:59320/collection1/ has no replicas
[junit4:junit4] 2> 69615 T320 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59320/collection1/
[junit4:junit4] 2> 69615 T320 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 69877 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 69897 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 69897 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 69897 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 69923 T320 oasc.ZkController.register We are http://127.0.0.1:59320/collection1/ and leader is http://127.0.0.1:59320/collection1/
[junit4:junit4] 2> 69923 T320 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59320
[junit4:junit4] 2> 69923 T320 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 69924 T320 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 69924 T320 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 69926 T320 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 69928 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 69928 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 69929 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 69995 T277 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 69997 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45263
[junit4:junit4] 2> 69999 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 69999 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 70000 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272
[junit4:junit4] 2> 70000 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/solr.xml
[junit4:junit4] 2> 70000 T277 oasc.CoreContainer.<init> New CoreContainer 768004907
[junit4:junit4] 2> 70001 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/'
[junit4:junit4] 2> 70001 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/'
[junit4:junit4] 2> 70048 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 70048 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 70048 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 70049 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 70049 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 70049 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 70050 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 70050 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 70050 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 70051 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 70054 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 70055 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
[junit4:junit4] 2> 70055 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 70056 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 70058 T333 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69b60e5d name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 70058 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 70061 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 70066 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 70068 T335 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71f2d728 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 70068 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 70073 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 71078 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45263_
[junit4:junit4] 2> 71079 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45263_
[junit4:junit4] 2> 71083 T306 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 71083 T299 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 71083 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 71083 T319 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 71083 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 71083 T335 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 71083 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 71089 T336 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 71090 T336 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 71409 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 71411 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:59320__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59320_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59320"}
[junit4:junit4] 2> 71415 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45263_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45263"}
[junit4:junit4] 2> 71416 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 71416 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 71447 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 71447 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 71447 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 71447 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 72091 T336 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1
[junit4:junit4] 2> 72091 T336 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 72092 T336 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 72092 T336 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 72094 T336 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1/'
[junit4:junit4] 2> 72094 T336 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1/lib/README' to classloader
[junit4:junit4] 2> 72095 T336 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 72122 T336 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 72156 T336 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 72157 T336 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 72162 T336 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 72514 T336 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 72521 T336 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 72523 T336 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 72539 T336 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 72543 T336 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 72546 T336 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 72548 T336 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 72548 T336 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 72548 T336 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 72550 T336 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 72550 T336 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 72550 T336 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 72550 T336 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371144523272/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/
[junit4:junit4] 2> 72550 T336 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
[junit4:junit4] 2> 72551 T336 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 72551 T336 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2
[junit4:junit4] 2> 72551 T336 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index/
[junit4:junit4] 2> 72552 T336 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 72552 T336 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index
[junit4:junit4] 2> 72598 T336 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 72598 T336 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 72601 T336 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 72601 T336 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 72601 T336 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 72602 T336 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 72602 T336 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 72602 T336 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 72603 T336 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 72603 T336 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 72604 T336 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 72606 T336 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 72609 T336 oass.SolrIndexSearcher.<init> Opening Searcher@2f828d24 main
[junit4:junit4] 2> 72610 T336 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 72610 T336 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 72614 T337 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2f828d24 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 72616 T336 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 72616 T336 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45263 collection:collection1 shard:shard2
[junit4:junit4] 2> 72617 T336 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 72624 T336 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 72627 T336 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 72627 T336 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 72627 T336 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45263/collection1/
[junit4:junit4] 2> 72628 T336 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 72628 T336 oasc.SyncStrategy.syncToMe http://127.0.0.1:45263/collection1/ has no replicas
[junit4:junit4] 2> 72628 T336 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45263/collection1/
[junit4:junit4] 2> 72628 T336 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 72956 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 72965 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 72966 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 72965 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 72966 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 72986 T336 oasc.ZkController.register We are http://127.0.0.1:45263/collection1/ and leader is http://127.0.0.1:45263/collection1/
[junit4:junit4] 2> 72986 T336 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45263
[junit4:junit4] 2> 72986 T336 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 72986 T336 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 72986 T336 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 72989 T336 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 72990 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 72991 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 72991 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 73067 T277 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 73070 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52190
[junit4:junit4] 2> 73070 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 73070 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 73071 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336
[junit4:junit4] 2> 73071 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/solr.xml
[junit4:junit4] 2> 73071 T277 oasc.CoreContainer.<init> New CoreContainer 985733305
[junit4:junit4] 2> 73072 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/'
[junit4:junit4] 2> 73072 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/'
[junit4:junit4] 2> 73118 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 73119 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 73119 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 73119 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 73119 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 73120 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 73120 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 73120 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 73121 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 73121 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 73125 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 73126 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
[junit4:junit4] 2> 73126 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 73127 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 73128 T349 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@534b79e name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 73129 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 73130 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 73134 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 73136 T351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35e47b48 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 73136 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 73140 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 74144 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52190_
[junit4:junit4] 2> 74146 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52190_
[junit4:junit4] 2> 74150 T319 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 74150 T335 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 74150 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 74150 T306 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 74150 T351 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 74151 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 74151 T299 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 74150 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 74151 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 74157 T352 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 74157 T352 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 74473 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 74475 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45263__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45263_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45263"}
[junit4:junit4] 2> 74479 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52190_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52190"}
[junit4:junit4] 2> 74479 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 74479 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 74484 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 74484 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 74485 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 74485 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 74485 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 75158 T352 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1
[junit4:junit4] 2> 75159 T352 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 75160 T352 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 75160 T352 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 75162 T352 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1/'
[junit4:junit4] 2> 75162 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1/lib/README' to classloader
[junit4:junit4] 2> 75163 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 75195 T352 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 75233 T352 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 75235 T352 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 75242 T352 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 75613 T352 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 75619 T352 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 75622 T352 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 75642 T352 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 75646 T352 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 75649 T352 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 75650 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 75651 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 75651 T352 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 75652 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 75652 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 75653 T352 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 75653 T352 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371144526336/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/
[junit4:junit4] 2> 75653 T352 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
[junit4:junit4] 2> 75654 T352 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 75654 T352 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3
[junit4:junit4] 2> 75654 T352 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/index/
[junit4:junit4] 2> 75654 T352 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 75655 T352 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/index
[junit4:junit4] 2> 75658 T352 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 75659 T352 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 75661 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 75661 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 75662 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 75662 T352 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 75663 T352 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 75663 T352 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 75663 T352 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 75664 T352 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 75664 T352 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 75666 T352 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 75669 T352 oass.SolrIndexSearcher.<init> Opening Searcher@aa21710 main
[junit4:junit4] 2> 75670 T352 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 75670 T352 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 75676 T353 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@aa21710 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 75678 T352 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 75678 T352 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52190 collection:collection1 shard:shard3
[junit4:junit4] 2> 75679 T352 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 75686 T352 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 75689 T352 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 75689 T352 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 75689 T352 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52190/collection1/
[junit4:junit4] 2> 75689 T352 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 75690 T352 oasc.SyncStrategy.syncToMe http://127.0.0.1:52190/collection1/ has no replicas
[junit4:junit4] 2> 75690 T352 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52190/collection1/
[junit4:junit4] 2> 75690 T352 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 75996 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 76005 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 76005 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 76005 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 76005 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 76005 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 76047 T352 oasc.ZkController.register We are http://127.0.0.1:52190/collection1/ and leader is http://127.0.0.1:52190/collection1/
[junit4:junit4] 2> 76047 T352 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52190
[junit4:junit4] 2> 76047 T352 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 76047 T352 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 76048 T352 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 76050 T352 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 76051 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 76051 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 76051 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 76118 T277 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 76120 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44393
[junit4:junit4] 2> 76121 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 76121 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 76121 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395
[junit4:junit4] 2> 76122 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/solr.xml
[junit4:junit4] 2> 76122 T277 oasc.CoreContainer.<init> New CoreContainer 925394222
[junit4:junit4] 2> 76122 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/'
[junit4:junit4] 2> 76122 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/'
[junit4:junit4] 2> 76164 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 76164 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 76164 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 76165 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 76165 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 76165 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 76166 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 76166 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 76166 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 76167 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 76170 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 76171 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
[junit4:junit4] 2> 76171 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 76172 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 76174 T365 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@145c1572 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 76174 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 76176 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 76180 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 76181 T367 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c066577 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 76181 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 76185 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 77190 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44393_
[junit4:junit4] 2> 77191 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44393_
[junit4:junit4] 2> 77193 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 77195 T299 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 77195 T335 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 77195 T367 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 77195 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 77195 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 77197 T319 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 77197 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 77198 T306 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 77198 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 77200 T351 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 77206 T368 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 77206 T368 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 77514 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 77515 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52190__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52190_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52190"}
[junit4:junit4] 2> 77521 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44393_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44393"}
[junit4:junit4] 2> 77521 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 77521 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 77527 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 77527 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 77527 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 77527 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 77527 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 77528 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 78207 T368 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1
[junit4:junit4] 2> 78208 T368 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 78209 T368 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 78209 T368 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 78211 T368 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1/'
[junit4:junit4] 2> 78211 T368 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1/lib/README' to classloader
[junit4:junit4] 2> 78212 T368 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 78248 T368 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 78285 T368 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 78287 T368 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 78291 T368 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 78608 T368 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 78615 T368 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 78617 T368 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 78638 T368 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 78643 T368 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 78648 T368 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 78650 T368 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 78650 T368 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 78651 T368 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 78652 T368 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 78652 T368 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 78653 T368 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 78653 T368 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371144529395/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/
[junit4:junit4] 2> 78653 T368 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
[junit4:junit4] 2> 78654 T368 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 78654 T368 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4
[junit4:junit4] 2> 78655 T368 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index/
[junit4:junit4] 2> 78655 T368 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 78656 T368 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index
[junit4:junit4] 2> 78659 T368 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 78659 T368 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 78661 T368 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 78661 T368 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 78662 T368 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 78663 T368 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 78663 T368 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 78663 T368 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 78664 T368 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 78664 T368 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 78665 T368 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 78668 T368 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 78673 T368 oass.SolrIndexSearcher.<init> Opening Searcher@78e556dd main
[junit4:junit4] 2> 78673 T368 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 78674 T368 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 78678 T369 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78e556dd main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 78680 T368 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 78680 T368 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44393 collection:collection1 shard:shard1
[junit4:junit4] 2> 78684 T368 oasc.ZkController.register We are http://127.0.0.1:44393/collection1/ and leader is http://127.0.0.1:59320/collection1/
[junit4:junit4] 2> 78684 T368 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44393
[junit4:junit4] 2> 78684 T368 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 78684 T368 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C57 name=collection1 org.apache.solr.core.SolrCore@d189531 url=http://127.0.0.1:44393/collection1 node=127.0.0.1:44393_ C57_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:44393_, base_url=http://127.0.0.1:44393}
[junit4:junit4] 2> 78687 T370 C57 P44393 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 78687 T368 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 78688 T370 C57 P44393 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 78689 T370 C57 P44393 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 78689 T370 C57 P44393 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 78689 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 78690 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 78690 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 78693 T370 C57 P44393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 78707 T312 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:44393__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 78768 T277 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 78771 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45641
[junit4:junit4] 2> 78771 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 78772 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 78772 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035
[junit4:junit4] 2> 78772 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/solr.xml
[junit4:junit4] 2> 78773 T277 oasc.CoreContainer.<init> New CoreContainer 1946744004
[junit4:junit4] 2> 78773 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/'
[junit4:junit4] 2> 78773 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/'
[junit4:junit4] 2> 78815 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 78816 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 78816 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 78816 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 78816 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 78817 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 78817 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 78817 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 78818 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 78818 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 78822 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 78822 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
[junit4:junit4] 2> 78823 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 78824 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 78825 T382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b69c717 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 78825 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 78835 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 78840 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 78841 T384 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2612fc39 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 78841 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 78848 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 79035 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 79036 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44393__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44393_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44393"}
[junit4:junit4] 2> 79040 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79041 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79041 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79041 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79040 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79041 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79041 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79708 T312 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:44393__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 79708 T312 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:44393__collection1&state=recovering&nodeName=127.0.0.1:44393_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
[junit4:junit4] 2> 79852 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45641_
[junit4:junit4] 2> 79853 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45641_
[junit4:junit4] 2> 79855 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79856 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79856 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 79856 T299 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 79856 T306 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 79856 T335 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 79857 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 79856 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 79857 T319 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 79856 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 79857 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 79859 T367 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 79859 T351 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 79859 T384 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 79864 T385 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 79865 T385 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 80552 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 80553 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45641_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45641"}
[junit4:junit4] 2> 80553 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 80553 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 80558 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 80558 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 80558 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 80558 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 80558 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 80558 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 80558 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 80866 T385 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1
[junit4:junit4] 2> 80866 T385 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 80867 T385 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 80868 T385 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 80869 T385 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1/'
[junit4:junit4] 2> 80870 T385 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1/lib/README' to classloader
[junit4:junit4] 2> 80870 T385 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 80894 T385 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 80922 T385 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 80924 T385 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 80928 T385 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 81248 T385 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 81254 T385 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 81256 T385 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 81272 T385 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 81276 T385 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 81279 T385 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 81280 T385 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 81281 T385 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 81281 T385 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 81282 T385 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 81282 T385 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 81283 T385 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 81283 T385 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371144532035/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/
[junit4:junit4] 2> 81283 T385 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
[junit4:junit4] 2> 81284 T385 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 81284 T385 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5
[junit4:junit4] 2> 81284 T385 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index/
[junit4:junit4] 2> 81285 T385 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 81285 T385 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index
[junit4:junit4] 2> 81288 T385 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 81288 T385 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 81290 T385 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 81291 T385 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 81291 T385 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 81292 T385 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 81292 T385 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 81292 T385 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 81293 T385 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 81293 T385 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 81294 T385 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 81296 T385 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 81299 T385 oass.SolrIndexSearcher.<init> Opening Searcher@250b2251 main
[junit4:junit4] 2> 81300 T385 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 81300 T385 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 81305 T386 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@250b2251 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 81307 T385 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 81307 T385 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45641 collection:collection1 shard:shard2
[junit4:junit4] 2> 81311 T385 oasc.ZkController.register We are http://127.0.0.1:45641/collection1/ and leader is http://127.0.0.1:45263/collection1/
[junit4:junit4] 2> 81311 T385 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45641
[junit4:junit4] 2> 81311 T385 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 81311 T385 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C58 name=collection1 org.apache.solr.core.SolrCore@2014e6de url=http://127.0.0.1:45641/collection1 node=127.0.0.1:45641_ C58_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:45641_, base_url=http://127.0.0.1:45641}
[junit4:junit4] 2> 81312 T387 C58 P45641 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 81312 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 81313 T387 C58 P45641 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 81313 T387 C58 P45641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 81313 T385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 81314 T387 C58 P45641 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 81315 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 81315 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 81316 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 81319 T330 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:45641__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 81396 T277 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 81399 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45982
[junit4:junit4] 2> 81400 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 81400 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 81400 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659
[junit4:junit4] 2> 81401 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/solr.xml
[junit4:junit4] 2> 81401 T277 oasc.CoreContainer.<init> New CoreContainer 243591166
[junit4:junit4] 2> 81402 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/'
[junit4:junit4] 2> 81402 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/'
[junit4:junit4] 2> 81449 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 81449 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 81450 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 81450 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 81450 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 81450 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 81451 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 81451 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 81451 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 81452 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 81457 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 81457 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
[junit4:junit4] 2> 81458 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 81459 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 81461 T399 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51e1ca54 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 81462 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 81464 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 81481 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 81481 T401 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@186a5fac name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 81482 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 81486 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C59 name=collection1 org.apache.solr.core.SolrCore@d189531 url=http://127.0.0.1:44393/collection1 node=127.0.0.1:44393_ C59_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:44393_, base_url=http://127.0.0.1:44393}
[junit4:junit4] 2> 81709 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59320/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 81714 T370 C59 P44393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 81721 T370 C59 P44393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44393 START replicas=[http://127.0.0.1:59320/collection1/] nUpdates=100
[junit4:junit4] 2> 81722 T370 C59 P44393 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 81722 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 81722 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 81722 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 81723 T370 C59 P44393 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 81723 T370 C59 P44393 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59320/collection1/. core=collection1
[junit4:junit4] 2> 81723 T370 C59 P44393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C60 name=collection1 org.apache.solr.core.SolrCore@7ff4ca39 url=http://127.0.0.1:59320/collection1 node=127.0.0.1:59320_ C60_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59320_, base_url=http://127.0.0.1:59320, leader=true}
[junit4:junit4] 2> 81728 T313 C60 P59320 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 81731 T315 C60 P59320 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 81733 T315 C60 P59320 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 81734 T315 C60 P59320 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 81746 T315 C60 P59320 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 81747 T315 C60 P59320 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 81748 T315 C60 P59320 oass.SolrIndexSearcher.<init> Opening Searcher@7f3220e3 realtime
[junit4:junit4] 2> 81748 T315 C60 P59320 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 81748 T315 C60 P59320 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> 81773 T370 C59 P44393 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 81774 T370 C59 P44393 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 81782 T370 C59 P44393 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 81785 T312 C60 P59320 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 81786 T312 C60 P59320 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 81787 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 81787 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 81787 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 81790 T313 C60 P59320 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 81790 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 81791 T370 C59 P44393 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131
[junit4:junit4] 2> 81792 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131 lockFactory=org.apache.lucene.store.NativeFSLockFactory@77347fda fullCopy=false
[junit4:junit4] 2> 81799 T314 C60 P59320 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=2
[junit4:junit4] 2> 81804 T370 C59 P44393 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 81807 T370 C59 P44393 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 81808 T370 C59 P44393 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 81812 T370 C59 P44393 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 81813 T370 C59 P44393 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 81813 T370 C59 P44393 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 81813 T370 C59 P44393 oass.SolrIndexSearcher.<init> Opening Searcher@1511258c main
[junit4:junit4] 2> 81815 T369 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1511258c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 81816 T370 C59 P44393 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131;done=true>>]
[junit4:junit4] 2> 81816 T370 C59 P44393 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131
[junit4:junit4] 2> 81816 T370 C59 P44393 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty4/index.20130614002855131
[junit4:junit4] 2> 81820 T370 C59 P44393 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 81820 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 81821 T370 C59 P44393 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 81821 T370 C59 P44393 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 81823 T370 C59 P44393 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 82065 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 82066 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45641__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45641_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45641"}
[junit4:junit4] 2> 82069 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44393__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44393_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44393"}
[junit4:junit4] 2> 82087 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82087 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82087 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82087 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82087 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82087 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82087 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82087 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82320 T330 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:45641__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 82320 T330 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:45641__collection1&state=recovering&nodeName=127.0.0.1:45641_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 82490 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45982_
[junit4:junit4] 2> 82491 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45982_
[junit4:junit4] 2> 82494 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82494 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82495 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82495 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 82495 T335 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 82495 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 82496 T299 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 82496 T306 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 82496 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 82496 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 82496 T319 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 82497 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 82498 T384 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 82499 T401 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 82499 T367 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 82499 T351 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 82506 T404 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 82506 T404 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 83598 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 83599 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45982_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45982"}
[junit4:junit4] 2> 83599 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 83599 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 83605 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 83605 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 83605 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 83605 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 83605 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 83605 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 83605 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 83605 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> C58_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:45641_, base_url=http://127.0.0.1:45641}
[junit4:junit4] 2> 84321 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45263/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 84322 T387 C58 P45641 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45641 START replicas=[http://127.0.0.1:45263/collection1/] nUpdates=100
[junit4:junit4] 2> 84322 T387 C58 P45641 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 84323 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 84323 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 84323 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 84324 T387 C58 P45641 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 84324 T387 C58 P45641 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45263/collection1/. core=collection1
[junit4:junit4] 2> 84324 T387 C58 P45641 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C61 name=collection1 org.apache.solr.core.SolrCore@4fd686fc url=http://127.0.0.1:45263/collection1 node=127.0.0.1:45263_ C61_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:45263_, base_url=http://127.0.0.1:45263, leader=true}
[junit4:junit4] 2> 84328 T330 C61 P45263 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 84330 T328 C61 P45263 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 84333 T328 C61 P45263 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 84335 T328 C61 P45263 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 84346 T328 C61 P45263 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 84347 T328 C61 P45263 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 84348 T328 C61 P45263 oass.SolrIndexSearcher.<init> Opening Searcher@2e358e22 realtime
[junit4:junit4] 2> 84348 T328 C61 P45263 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 84348 T328 C61 P45263 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> 84349 T387 C58 P45641 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 84350 T387 C58 P45641 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 84352 T331 C61 P45263 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 84353 T331 C61 P45263 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 84353 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 84353 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 84354 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 84356 T330 C61 P45263 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 84357 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 84357 T387 C58 P45641 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697
[junit4:junit4] 2> 84358 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697 lockFactory=org.apache.lucene.store.NativeFSLockFactory@562cf72d fullCopy=false
[junit4:junit4] 2> 84361 T328 C61 P45263 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 84362 T387 C58 P45641 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 84367 T387 C58 P45641 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 84367 T387 C58 P45641 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 84370 T387 C58 P45641 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 84370 T387 C58 P45641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 84371 T387 C58 P45641 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 84371 T387 C58 P45641 oass.SolrIndexSearcher.<init> Opening Searcher@33c346d7 main
[junit4:junit4] 2> 84372 T386 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33c346d7 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 84373 T387 C58 P45641 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697;done=true>>]
[junit4:junit4] 2> 84373 T387 C58 P45641 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697
[junit4:junit4] 2> 84373 T387 C58 P45641 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty5/index.20130614002857697
[junit4:junit4] 2> 84373 T387 C58 P45641 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 84373 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 84374 T387 C58 P45641 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 84374 T387 C58 P45641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 84376 T387 C58 P45641 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 84508 T404 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1
[junit4:junit4] 2> 84508 T404 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 84509 T404 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 84509 T404 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 84510 T404 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1/'
[junit4:junit4] 2> 84511 T404 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1/lib/README' to classloader
[junit4:junit4] 2> 84511 T404 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 84539 T404 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 84568 T404 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 84570 T404 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 84574 T404 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 84915 T404 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 84922 T404 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 84924 T404 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 84942 T404 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 84947 T404 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 84952 T404 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 84954 T404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 84954 T404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 84955 T404 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 84956 T404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 84956 T404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 84957 T404 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 84957 T404 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371144534659/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/
[junit4:junit4] 2> 84957 T404 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
[junit4:junit4] 2> 84958 T404 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 84959 T404 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6
[junit4:junit4] 2> 84959 T404 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/index/
[junit4:junit4] 2> 84959 T404 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 84960 T404 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/index
[junit4:junit4] 2> 84978 T404 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 84979 T404 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 84981 T404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 84981 T404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 84982 T404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 84982 T404 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 84983 T404 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 84983 T404 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 84986 T404 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 84986 T404 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 84987 T404 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 84989 T404 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 84992 T404 oass.SolrIndexSearcher.<init> Opening Searcher@7c14f9fd main
[junit4:junit4] 2> 84992 T404 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 84993 T404 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 84997 T407 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c14f9fd main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 84999 T404 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 84999 T404 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45982 collection:collection1 shard:shard3
[junit4:junit4] 2> 85003 T404 oasc.ZkController.register We are http://127.0.0.1:45982/collection1/ and leader is http://127.0.0.1:52190/collection1/
[junit4:junit4] 2> 85003 T404 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45982
[junit4:junit4] 2> 85003 T404 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 85003 T404 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C62 name=collection1 org.apache.solr.core.SolrCore@2ce806a3 url=http://127.0.0.1:45982/collection1 node=127.0.0.1:45982_ C62_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:45982_, base_url=http://127.0.0.1:45982}
[junit4:junit4] 2> 85004 T408 C62 P45982 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 85004 T404 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 85004 T408 C62 P45982 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 85005 T408 C62 P45982 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 85005 T408 C62 P45982 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 85006 T277 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 85006 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 85007 T408 C62 P45982 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 85007 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 85012 T347 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:45982__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 85099 T277 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 85102 T277 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41375
[junit4:junit4] 2> 85102 T277 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 85102 T277 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 85103 T277 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352
[junit4:junit4] 2> 85103 T277 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/solr.xml
[junit4:junit4] 2> 85103 T277 oasc.CoreContainer.<init> New CoreContainer 28452598
[junit4:junit4] 2> 85104 T277 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/'
[junit4:junit4] 2> 85104 T277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/'
[junit4:junit4] 2> 85112 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 85113 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45641__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45641_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45641"}
[junit4:junit4] 2> 85116 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45982__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45982_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45982"}
[junit4:junit4] 2> 85121 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 85121 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 85121 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 85122 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 85121 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 85121 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 85121 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 85122 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 85152 T277 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 85153 T277 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 85153 T277 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 85153 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 85154 T277 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 85154 T277 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 85154 T277 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 85155 T277 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 85155 T277 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 85155 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 85159 T277 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 85160 T277 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38719/solr
[junit4:junit4] 2> 85160 T277 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 85161 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 85163 T420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46464845 name:ZooKeeperConnection Watcher:127.0.0.1:38719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 85163 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 85164 T277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 85169 T277 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 85170 T422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ab27676 name:ZooKeeperConnection Watcher:127.0.0.1:38719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 85171 T277 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 85175 T277 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 86013 T347 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:45982__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 86013 T347 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:45982__collection1&state=recovering&nodeName=127.0.0.1:45982_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 86179 T277 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41375_
[junit4:junit4] 2> 86181 T277 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41375_
[junit4:junit4] 2> 86183 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 86184 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 86184 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 86184 T306 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 86184 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 86185 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86185 T335 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 86185 T299 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 86185 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86185 T319 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 86186 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86185 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86186 T422 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 86188 T367 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 86188 T384 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 86187 T401 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 86188 T351 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 86193 T423 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 86193 T423 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 86630 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 86631 T300 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41375_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41375"}
[junit4:junit4] 2> 86631 T300 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 86632 T300 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 86637 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86637 T384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86637 T319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86637 T422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86637 T401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86637 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86637 T335 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86637 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 86637 T367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 87195 T423 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1
[junit4:junit4] 2> 87195 T423 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 87196 T423 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 87196 T423 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 87198 T423 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1/'
[junit4:junit4] 2> 87198 T423 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1/lib/README' to classloader
[junit4:junit4] 2> 87199 T423 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 87228 T423 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 87258 T423 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 87260 T423 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 87264 T423 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 87620 T423 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 87627 T423 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 87629 T423 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 87648 T423 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 87651 T423 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 87655 T423 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 87657 T423 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 87657 T423 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 87657 T423 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 87658 T423 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 87659 T423 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 87659 T423 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 87659 T423 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371144538352/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/
[junit4:junit4] 2> 87659 T423 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4be44cc9
[junit4:junit4] 2> 87660 T423 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 87660 T423 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7
[junit4:junit4] 2> 87661 T423 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/index/
[junit4:junit4] 2> 87661 T423 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 87661 T423 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/index
[junit4:junit4] 2> 87664 T423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371144516371/jetty7/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 87664 T423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 87666 T423 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 87667 T423 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 87667 T423 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 87668 T423 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 87668 T423 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 87668 T423 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 87669 T423 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 87669 T423 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 87670 T423 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 87672 T423 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 87675 T423 oass.SolrIndexSearcher.<init> Opening Searcher@3856fa9e main
[junit4:junit4] 2> 87675 T423 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 87676 T423 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 87682 T424 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3856fa9e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 87685 T423 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 87685 T423 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41375 collection:collectio
[...truncated too long message...]
OrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 366507 T1266 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 366507 T1266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 366507 T1266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 366507 T1266 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 366508 T1266 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr
[junit4:junit4] 2> 366508 T1266 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/'
[junit4:junit4] 2> 366516 T1266 oasc.CoreContainer.<init> New CoreContainer 1238919290
[junit4:junit4] 2> 366541 T1266 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 366541 T1266 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/
[junit4:junit4] 2> 366541 T1266 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4] 2> 366542 T1266 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 366542 T1266 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357
[junit4:junit4] 2> 366542 T1266 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index/
[junit4:junit4] 2> 366543 T1266 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 366543 T1266 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index
[junit4:junit4] 2> 366544 T1266 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5510ce87 lockFactory=org.apache.lucene.store.NativeFSLockFactory@68c9703a),segFN=segments_1,generation=1}
[junit4:junit4] 2> 366545 T1266 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 366546 T1266 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 366598 T1266 oass.SolrIndexSearcher.<init> Opening Searcher@67713565 main
[junit4:junit4] 2> 366598 T1266 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 366599 T1266 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 366599 T1266 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 366599 T1267 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67713565 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 366601 T1266 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4] 2> 366605 T1266 oas.SolrTestCaseJ4.setUp ###Starting testTieredMergePolicyConfig
[junit4:junit4] 2> 366605 T1266 oas.SolrTestCaseJ4.tearDown ###Ending testTieredMergePolicyConfig
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestMergePolicyConfig -Dtests.method=testTieredMergePolicyConfig -Dtests.seed=42BD083CD5E13AD1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=no -Dtests.timezone=America/Noronha -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 0.01s J1 | TestMergePolicyConfig.testTieredMergePolicyConfig <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<0.0> but was:<1.0>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([42BD083CD5E13AD1:E04515B88B8CCCBA]:0)
[junit4:junit4] > at org.apache.solr.core.TestMergePolicyConfig.testTieredMergePolicyConfig(TestMergePolicyConfig.java:43)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 366610 T1266 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 366610 T1266 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1238919290
[junit4:junit4] 2> 366610 T1266 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@6fc0e4e3
[junit4:junit4] 2> 366610 T1266 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 366611 T1266 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 366611 T1266 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 366611 T1266 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 366611 T1266 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 366612 T1266 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 366612 T1266 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357;done=false>>]
[junit4:junit4] 2> 366612 T1266 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357
[junit4:junit4] 2> 366612 T1266 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index;done=false>>]
[junit4:junit4] 2> 366613 T1266 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-TestMergePolicyConfig-1371144819357/index
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {}, docValues:{}, sim=DefaultSimilarity, locale=no, timezone=America/Noronha
[junit4:junit4] 2> NOTE: Linux 3.2.0-45-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=1,free=373635368,total=521142272
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestBM25SimilarityFactory, HighlighterConfigTest, TestDistributedSearch, TestSchemaNameResource, DefaultValueUpdateProcessorTest, SolrIndexConfigTest, TestStressLucene, TestSolrXMLSerializer, TestOmitPositions, JSONWriterTest, TestBinaryResponseWriter, TestLMJelinekMercerSimilarityFactory, TestFieldResource, ClusterStateTest, TestCopyFieldCollectionResource, TestRangeQuery, ChaosMonkeySafeLeaderTest, TestFieldTypeCollectionResource, TestSolrJ, NotRequiredUniqueKeyTest, DocumentAnalysisRequestHandlerTest, PathHierarchyTokenizerFactoryTest, IndexBasedSpellCheckerTest, SolrRequestParserTest, TestCSVResponseWriter, BasicDistributedZk2Test, PingRequestHandlerTest, MultiTermTest, SortByFunctionTest, TestDistributedGrouping, StandardRequestHandlerTest, OutputWriterTest, RequiredFieldsTest, TestIndexingPerformance, TestMultiCoreConfBootstrap, SuggesterWFSTTest, TestSolrQueryParserDefaultOperatorResource, CurrencyFieldXmlFileTest, DateMathParserTest, TestAtomicUpdateErrorCases, SolrCoreCheckLockOnStartupTest, TestDefaultSimilarityFactory, TestComponentsName, TestDynamicFieldCollectionResource, TestMergePolicyConfig]
[junit4:junit4] Completed on J1 in 0.61s, 1 test, 1 failure <<< FAILURES!
[...truncated 565 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:385: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:365: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1240: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 2 failures, 13 ignored (7 assumptions)
Total time: 39 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure