You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/01/12 09:46:41 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_25) - Build #
11579 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11579/
Java: 32bit/jdk1.8.0_25 -server -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.util.SimplePostToolTest.testTypeSupported
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([31787CCB795F3064:A5CC4A286E60910B]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.util.SimplePostToolTest.testTypeSupported(SimplePostToolTest.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 9473 lines...]
[junit4] Suite: org.apache.solr.util.SimplePostToolTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.util.SimplePostToolTest 31787CCB795F3064-001/init-core-data-001
[junit4] 2> 1247217 T6934 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 1247220 T6934 oas.SolrTestCaseJ4.setUp ###Starting testAppendUrlPath
[junit4] 2> 1247232 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testAppendUrlPath
[junit4] 2> 1247237 T6934 oas.SolrTestCaseJ4.setUp ###Starting testTypeSupported
[junit4] 2> 1247239 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testTypeSupported
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SimplePostToolTest -Dtests.method=testTypeSupported -Dtests.seed=31787CCB795F3064 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh_TW -Dtests.timezone=Asia/Anadyr -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 0.04s J1 | SimplePostToolTest.testTypeSupported <<<
[junit4] > Throwable #1: java.lang.AssertionError
[junit4] > at __randomizedtesting.SeedInfo.seed([31787CCB795F3064:A5CC4A286E60910B]:0)
[junit4] > at org.apache.solr.util.SimplePostToolTest.testTypeSupported(SimplePostToolTest.java:116)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1247281 T6934 oas.SolrTestCaseJ4.setUp ###Starting testDoFilesMode
[junit4] 1> Indexing directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/exampledocs (2 files, depth=0)
[junit4] 1> POSTing file example.html (text/html) to [base]/extract
[junit4] 1> POSTing file example.txt (text/plain) to [base]/extract
[junit4] 2> 1247284 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testDoFilesMode
[junit4] 2> 1247288 T6934 oas.SolrTestCaseJ4.setUp ###Starting testNormalizeUrlEnding
[junit4] 2> 1247289 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testNormalizeUrlEnding
[junit4] 2> 1247293 T6934 oas.SolrTestCaseJ4.setUp ###Starting testDoWebMode
[junit4] 1> Entering crawl at level 0 (1 links total, 1 new)
[junit4] 1> POSTed web resource http://[ff01::114] (depth: 0)
[junit4] 1> Entering crawl at level 1 (2 links total, 2 new)
[junit4] 1> POSTed web resource http://[ff01::114]/page1 (depth: 1)
[junit4] 1> POSTed web resource http://[ff01::114]/page2 (depth: 1)
[junit4] 1> Entering crawl at level 2 (2 links total, 2 new)
[junit4] 1> POSTed web resource http://[ff01::114]/page1/foo (depth: 2)
[junit4] 2> SimplePostTool: WARNING: The URL http://[ff01::114]/disallowed returned a HTTP result status of 403
[junit4] 1> Entering crawl at level 3 (1 links total, 1 new)
[junit4] 1> POSTed web resource http://[ff01::114]/page1/foo/bar (depth: 3)
[junit4] 1> Entering crawl at level 0 (1 links total, 1 new)
[junit4] 1> POSTed web resource http://[ff01::114] (depth: 0)
[junit4] 1> Entering crawl at level 1 (2 links total, 2 new)
[junit4] 1> POSTed web resource http://[ff01::114]/page1 (depth: 1)
[junit4] 1> POSTed web resource http://[ff01::114]/page2 (depth: 1)
[junit4] 1> Entering crawl at level 0 (1 links total, 1 new)
[junit4] 1> POSTed web resource http://[ff01::114] (depth: 0)
[junit4] 1> Entering crawl at level 1 (2 links total, 2 new)
[junit4] 1> POSTed web resource http://[ff01::114]/page1 (depth: 1)
[junit4] 1> POSTed web resource http://[ff01::114]/page2 (depth: 1)
[junit4] 1> Entering crawl at level 2 (2 links total, 2 new)
[junit4] 1> POSTed web resource http://[ff01::114]/page1/foo (depth: 2)
[junit4] 1> POSTed web resource http://[ff01::114]/disallowed (depth: 2)
[junit4] 1> Entering crawl at level 3 (1 links total, 1 new)
[junit4] 1> POSTed web resource http://[ff01::114]/page1/foo/bar (depth: 3)
[junit4] 2> 1247301 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testDoWebMode
[junit4] 2> 1247305 T6934 oas.SolrTestCaseJ4.setUp ###Starting testAppendParam
[junit4] 2> 1247306 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testAppendParam
[junit4] 2> 1247310 T6934 oas.SolrTestCaseJ4.setUp ###Starting testParseArgsAndInit
[junit4] 2> 1247312 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testParseArgsAndInit
[junit4] 2> 1247316 T6934 oas.SolrTestCaseJ4.setUp ###Starting testRobotsExclusion
[junit4] 2> 1247317 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testRobotsExclusion
[junit4] 2> 1247322 T6934 oas.SolrTestCaseJ4.setUp ###Starting testGuessType
[junit4] 2> 1247324 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testGuessType
[junit4] 2> 1247329 T6934 oas.SolrTestCaseJ4.setUp ###Starting testIsOn
[junit4] 2> 1247331 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testIsOn
[junit4] 2> 1247336 T6934 oas.SolrTestCaseJ4.setUp ###Starting testComputeFullUrl
[junit4] 2> 1247339 T6934 oas.SolrTestCaseJ4.tearDown ###Ending testComputeFullUrl
[junit4] 2> 1247342 T6934 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.util.SimplePostToolTest 31787CCB795F3064-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=zh_TW, timezone=Asia/Anadyr
[junit4] 2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_25 (32-bit)/cpus=8,threads=1,free=143181424,total=304676864
[junit4] 2> NOTE: All tests run in this JVM: [TermVectorComponentTest, SolrCmdDistributorTest, StandardRequestHandlerTest, ReplicationFactorTest, TestFoldingMultitermQuery, HdfsBasicDistributedZk2Test, IndexBasedSpellCheckerTest, IndexSchemaTest, TestNonDefinedSimilarityFactory, ZkNodePropsTest, TestExactStatsCache, TestSolrDeletionPolicy1, QueryEqualityTest, DistribCursorPagingTest, DocValuesMissingTest, FullSolrCloudDistribCmdsTest, TestSchemaResource, TestDynamicFieldResource, DistributedQueueTest, TestManagedResource, CopyFieldTest, MoreLikeThisHandlerTest, TestBulkSchemaAPI, TestDefaultStatsCache, TestLMJelinekMercerSimilarityFactory, TermsComponentTest, ShardSplitTest, AnalyticsMergeStrategyTest, TestConfig, PolyFieldTest, SpellCheckComponentTest, DistributedQueryComponentOptimizationTest, BasicZkTest, TestSolr4Spatial, SchemaVersionSpecificBehaviorTest, OverseerCollectionProcessorTest, TestReversedWildcardFilterFactory, LeaderElectionIntegrationTest, BlockDirectoryTest, AnalyticsQueryTest, TestCloudSchemaless, TestCoreDiscovery, BasicFunctionalityTest, DisMaxRequestHandlerTest, PreAnalyzedUpdateProcessorTest, TestSolrQueryParser, StatelessScriptUpdateProcessorFactoryTest, TestSolrConfigHandlerCloud, JsonLoaderTest, BadIndexSchemaTest, CoreAdminCreateDiscoverTest, ResponseHeaderTest, ScriptEngineTest, TestLFUCache, TestExactSharedStatsCache, CurrencyFieldOpenExchangeTest, ReturnFieldsTest, CurrencyFieldXmlFileTest, SpellPossibilityIteratorTest, VMParamsZkACLAndCredentialsProvidersTest, TestMiniSolrCloudCluster, EchoParamsTest, TestDynamicFieldCollectionResource, DistributedExpandComponentTest, TestSolrIndexConfig, SimpleMLTQParserTest, TestInitParams, TestManagedStopFilterFactory, TestDistributedGrouping, SynonymTokenizerTest, TestRangeQuery, TestFieldTypeCollectionResource, BadComponentTest, TestConfigSets, TestLRUStatsCache, TestSerializedLuceneMatchVersion, TestPostingsSolrHighlighter, LoggingHandlerTest, CSVRequestHandlerTest, TestBM25SimilarityFactory, SuggesterTSTTest, TestElisionMultitermQuery, SOLR749Test, ConnectionManagerTest, CachingDirectoryFactoryTest, QueryParsingTest, AnalysisErrorHandlingTest, AutoCommitTest, ConvertedLegacyTest, FieldMutatingUpdateProcessorTest, TestReplicationHandlerBackup, TestConfigReload, WordBreakSolrSpellCheckerTest, TestCharFilters, FileBasedSpellCheckerTest, DistanceUnitsTest, SpatialFilterTest, XsltUpdateRequestHandlerTest, TestJmxMonitoredMap, DirectSolrSpellCheckerTest, DistribDocExpirationUpdateProcessorTest, TestHighlightDedupGrouping, RegexBoostProcessorTest, BasicDistributedZk2Test, TestFieldSortValues, DistributedQueryComponentCustomSortTest, ChaosMonkeySafeLeaderTest, UUIDUpdateProcessorFallbackTest, TestBulkSchemaConcurrent, SyncSliceTest, ClusterStateUpdateTest, ZkSolrClientTest, ShardRoutingCustomTest, TestFaceting, TestRecovery, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, ZkControllerTest, TestRealTimeGet, TestJoin, HardAutoCommitTest, DistributedTermsComponentTest, TestCoreContainer, TestGroupingSearch, TestBadConfig, HighlighterTest, ShowFileRequestHandlerTest, SimplePostToolTest]
[junit4] Completed on J1 in 0.15s, 11 tests, 1 failure <<< FAILURES!
[...truncated 602 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 461 suites, 1870 tests, 1 failure, 47 ignored (22 assumptions)
Total time: 61 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_25 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_40-ea-b20) -
Build # 11580 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11580/
Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseParallelGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([273BBFE4963E4923]:0)
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([273BBFE4963E4923]:0)
Build Log:
[...truncated 10184 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/init-core-data-001
[junit4] 2> 881492 T6375 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 881493 T6375 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 881495 T6375 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 881495 T6375 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 881496 T6376 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 881596 T6375 oasc.ZkTestServer.run start zk server on port:54337
[junit4] 2> 881596 T6375 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 881597 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 881599 T6383 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7057aa1 name:ZooKeeperConnection Watcher:127.0.0.1:54337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 881599 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 881600 T6375 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 881600 T6375 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 881602 T6375 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 881603 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 881604 T6386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3639c58f name:ZooKeeperConnection Watcher:127.0.0.1:54337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 881604 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 881604 T6375 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 881605 T6375 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 881606 T6375 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 881607 T6375 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 881607 T6375 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 881608 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 881609 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 881610 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 881610 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 881611 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 881612 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 881612 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 881613 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 881614 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 881614 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 881615 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 881615 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 881616 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 881616 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 881617 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 881617 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 881618 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 881618 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 881619 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 881620 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 881621 T6375 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 881621 T6375 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 881622 T6375 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 881622 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 881623 T6389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6007bbc4 name:ZooKeeperConnection Watcher:127.0.0.1:54337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 881623 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 881624 T6375 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 881859 T6375 oejs.Server.doStart jetty-9.2.6.v20141205
[junit4] 2> 881877 T6375 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@3e9bc11c{/,null,AVAILABLE}
[junit4] 2> 881887 T6375 oejs.AbstractConnector.doStart Started ServerConnector@3e52f865{SSL-http/1.1}{127.0.0.1:39766}
[junit4] 2> 881888 T6375 oejs.Server.doStart Started @882916ms
[junit4] 2> 881889 T6375 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 881890 T6375 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 881890 T6375 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-002
[junit4] 2> 881890 T6375 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-002/'
[junit4] 2> 881902 T6375 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-002/solr.xml
[junit4] 2> 881910 T6375 oasc.CoreContainer.<init> New CoreContainer 943786978
[junit4] 2> 881910 T6375 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-002/]
[junit4] 2> 881911 T6375 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 881911 T6375 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 881912 T6375 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 881912 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 881912 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 881912 T6375 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 881913 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 881913 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 881913 T6375 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 881914 T6375 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 881915 T6375 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 881916 T6375 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 881916 T6375 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 881916 T6375 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 881916 T6375 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54337/solr
[junit4] 2> 881917 T6375 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 881917 T6375 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 881917 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 881919 T6403 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54717786 name:ZooKeeperConnection Watcher:127.0.0.1:54337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 881919 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 881920 T6375 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 881921 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 881922 T6406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3735015e name:ZooKeeperConnection Watcher:127.0.0.1:54337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 881922 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 881923 T6375 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 881925 T6375 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 881926 T6375 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 881928 T6375 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 881929 T6375 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 881930 T6375 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 881931 T6375 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39766_
[junit4] 2> 881932 T6375 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39766_
[junit4] 2> 881933 T6375 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 881934 T6375 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 881935 T6375 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 881935 T6375 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39766_
[junit4] 2> 881936 T6375 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 881936 T6375 oasc.Overseer.start Overseer (id=93130563923804164-127.0.0.1:39766_-n_0000000000) starting
[junit4] 2> 881938 T6375 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 881941 T6375 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 881942 T6408 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 881942 T6375 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 881943 T6375 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 881944 T6375 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 881945 T6407 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 881946 T6410 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 881947 T6410 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 881947 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 881947 T6410 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 881949 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:39766",
[junit4] 2> "node_name":"127.0.0.1:39766_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 881949 T6407 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:39766",
[junit4] 2> "node_name":"127.0.0.1:39766_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 881949 T6407 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
[junit4] 2> 881950 T6407 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 881950 T6406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 882948 T6410 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 882948 T6410 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 882949 T6410 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 882949 T6410 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 882949 T6410 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 882950 T6410 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-002/collection1/'
[junit4] 2> 882950 T6410 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest%20273BBFE4963E4923-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 882950 T6410 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest%20273BBFE4963E4923-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 882950 T6410 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest%20273BBFE4963E4923-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 882959 T6410 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
[junit4] 2> 882961 T6410 oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 882964 T6410 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 882968 T6410 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 882977 T6410 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 882978 T6410 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 882981 T6410 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 883063 T6410 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 883065 T6410 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 883066 T6410 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 883068 T6410 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 883076 T6410 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 883076 T6410 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 883077 T6410 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/control/data/
[junit4] 2> 883077 T6410 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@615e5c33
[junit4] 2> 883078 T6410 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/control/data
[junit4] 2> 883078 T6410 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/control/data/index/
[junit4] 2> 883078 T6410 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 883078 T6410 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/control/data/index
[junit4] 2> 883079 T6410 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=4.6474609375, floorSegmentMB=2.1572265625, forceMergeDeletesPctAllowed=9.000088723067844, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7718117633583595
[junit4] 2> 883157 T6410 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 883157 T6410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 883159 T6410 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 883159 T6410 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 883159 T6410 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 883159 T6410 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 883160 T6410 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 883160 T6410 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 883160 T6410 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 883160 T6410 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 883160 T6410 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 883161 T6410 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 883161 T6410 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 883161 T6410 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 883161 T6410 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 883161 T6410 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
[junit4] 2> 883162 T6410 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
[junit4] 2> 883162 T6410 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
[junit4] 2> 883162 T6410 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
[junit4] 2> 883162 T6410 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
[junit4] 2> 883163 T6410 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
[junit4] 2> 883163 T6410 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
[junit4] 2> 883163 T6410 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
[junit4] 2> 883163 T6410 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
[junit4] 2> 883164 T6410 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
[junit4] 2> 883164 T6410 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
[junit4] 2> 883164 T6410 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
[junit4] 2> 883164 T6410 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 883165 T6410 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 883166 T6410 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 883166 T6410 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 883167 T6410 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 883168 T6410 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 883168 T6410 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 883169 T6410 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 883169 T6410 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 883169 T6410 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 883170 T6410 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 883170 T6410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 883170 T6410 oass.SolrIndexSearcher.<init> Opening Searcher@5ab02aac[collection1] main
[junit4] 2> 883170 T6410 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 883171 T6410 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 883171 T6410 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 883172 T6410 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 883172 T6410 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 883172 T6410 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 883173 T6410 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 883173 T6410 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 883173 T6410 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 883173 T6410 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 883174 T6411 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5ab02aac[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 883174 T6410 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 883174 T6414 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:39766 collection:control_collection shard:shard1
[junit4] 2> 883175 T6375 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 883175 T6375 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 883175 T6414 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 883177 T6375 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 883177 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 883178 T6414 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 883178 T6417 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@581db93b name:ZooKeeperConnection Watcher:127.0.0.1:54337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 883179 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 883179 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 883179 T6375 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 883179 T6414 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 883179 T6414 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1006 name=collection1 org.apache.solr.core.SolrCore@25a03f08 url=https://127.0.0.1:39766/collection1 node=127.0.0.1:39766_ C1006_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:39766, node_name=127.0.0.1:39766_, state=down}
[junit4] 2> 883180 T6414 C1006 P39766 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:39766/collection1/
[junit4] 2> 883180 T6414 C1006 P39766 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 883180 T6414 C1006 P39766 oasc.SyncStrategy.syncToMe https://127.0.0.1:39766/collection1/ has no replicas
[junit4] 2> 883180 T6414 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:39766/collection1/ shard1
[junit4] 2> 883180 T6414 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 883180 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection"} current state version: 1
[junit4] 2> 883181 T6375 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 883182 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 883183 T6375 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 883183 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"https://127.0.0.1:39766",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 883183 T6375 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
[junit4] 2> 883183 T6375 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 883184 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 883185 T6420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@391af239 name:ZooKeeperConnection Watcher:127.0.0.1:54337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 883185 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 883185 T6375 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 883186 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 883187 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"create",
[junit4] 2> "name":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "stateFormat":2} current state version: 1
[junit4] 2> 883187 T6407 oasco.ClusterStateMutator.createCollection building a new cName: collection1
[junit4] 2> 883187 T6417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 883187 T6406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 883291 T6407 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
[junit4] 2> 883291 T6417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 883333 T6414 oasc.ZkController.register We are https://127.0.0.1:39766/collection1/ and leader is https://127.0.0.1:39766/collection1/
[junit4] 2> 883334 T6414 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:39766
[junit4] 2> 883334 T6414 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 883334 T6414 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 883334 T6414 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 883335 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 883336 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:39766",
[junit4] 2> "node_name":"127.0.0.1:39766_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"} current state version: 3
[junit4] 2> 883336 T6407 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:39766",
[junit4] 2> "node_name":"127.0.0.1:39766_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 883424 T6375 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 883425 T6375 oejs.Server.doStart jetty-9.2.6.v20141205
[junit4] 2> 883426 T6375 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@5e88206a{/,null,AVAILABLE}
[junit4] 2> 883429 T6375 oejs.AbstractConnector.doStart Started ServerConnector@51925adc{SSL-http/1.1}{127.0.0.1:43219}
[junit4] 2> 883429 T6375 oejs.Server.doStart Started @884457ms
[junit4] 2> 883430 T6375 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 883430 T6375 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 883430 T6375 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-003
[junit4] 2> 883431 T6375 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-003/'
[junit4] 2> 883438 T6406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 883438 T6417 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 883442 T6375 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-003/solr.xml
[junit4] 2> 883456 T6375 oasc.CoreContainer.<init> New CoreContainer 236040776
[junit4] 2> 883457 T6375 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-003/]
[junit4] 2> 883457 T6375 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 883457 T6375 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 883457 T6375 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 883458 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 883458 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 883458 T6375 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 883458 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 883458 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 883459 T6375 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 883459 T6375 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 883460 T6375 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 883461 T6375 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 883461 T6375 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 883461 T6375 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 883461 T6375 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54337/solr
[junit4] 2> 883462 T6375 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 883462 T6375 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 883462 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 883464 T6434 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34990251 name:ZooKeeperConnection Watcher:127.0.0.1:54337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 883464 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 883464 T6375 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 883465 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 883466 T6437 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28f5d3de name:ZooKeeperConnection Watcher:127.0.0.1:54337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 883467 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 883470 T6375 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 884473 T6375 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43219_
[junit4] 2> 884474 T6375 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43219_
[junit4] 2> 884476 T6375 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 884478 T6438 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 884478 T6438 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 884479 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 884479 T6438 oasc.ZkController.preRegister Registering watch for external collection collection1
[junit4] 2> 884480 T6438 oascc.ZkStateReader.addZkWatch addZkWatch collection1
[junit4] 2> 884480 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:43219",
[junit4] 2> "node_name":"127.0.0.1:43219_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"} current state version: 4
[junit4] 2> 884480 T6438 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper...
[junit4] 2> 884480 T6407 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:43219",
[junit4] 2> "node_name":"127.0.0.1:43219_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 884480 T6407 oasco.ReplicaMutator.updateState Collection already exists with numShards=1
[junit4] 2> 884481 T6407 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 884481 T6438 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0
[junit4] 2> 884481 T6438 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 884582 T6407 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
[junit4] 2> 884583 T6437 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 884584 T6437 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1
[junit4] 2> 885481 T6438 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 885482 T6438 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 885482 T6438 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 885482 T6438 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 885483 T6438 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 885483 T6438 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-003/collection1/'
[junit4] 2> 885483 T6438 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest%20273BBFE4963E4923-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 885484 T6438 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest%20273BBFE4963E4923-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 885484 T6438 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest%20273BBFE4963E4923-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 885493 T6438 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
[junit4] 2> 885494 T6438 oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 885499 T6438 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 885504 T6438 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 885516 T6438 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 885517 T6438 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 885521 T6438 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 885607 T6438 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 885609 T6438 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 885609 T6438 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 885611 T6438 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 885621 T6438 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 885621 T6438 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 885621 T6438 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty1/
[junit4] 2> 885622 T6438 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@615e5c33
[junit4] 2> 885622 T6438 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty1
[junit4] 2> 885622 T6438 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty1/index/
[junit4] 2> 885623 T6438 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 885623 T6438 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty1/index
[junit4] 2> 885623 T6438 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=4.6474609375, floorSegmentMB=2.1572265625, forceMergeDeletesPctAllowed=9.000088723067844, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7718117633583595
[junit4] 2> 885630 T6438 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 885631 T6438 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 885633 T6438 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 885633 T6438 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 885633 T6438 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 885633 T6438 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 885633 T6438 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 885634 T6438 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 885634 T6438 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 885634 T6438 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 885634 T6438 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 885634 T6438 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 885635 T6438 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 885635 T6438 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 885635 T6438 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 885635 T6438 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
[junit4] 2> 885636 T6438 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
[junit4] 2> 885637 T6438 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
[junit4] 2> 885637 T6438 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
[junit4] 2> 885638 T6438 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
[junit4] 2> 885638 T6438 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
[junit4] 2> 885639 T6438 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
[junit4] 2> 885639 T6438 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
[junit4] 2> 885640 T6438 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
[junit4] 2> 885640 T6438 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
[junit4] 2> 885641 T6438 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
[junit4] 2> 885641 T6438 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
[junit4] 2> 885641 T6438 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 885642 T6438 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 885644 T6438 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 885645 T6438 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 885646 T6438 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 885647 T6438 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 885647 T6438 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 885648 T6438 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 885648 T6438 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 885650 T6438 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 885650 T6438 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 885651 T6438 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 885651 T6438 oass.SolrIndexSearcher.<init> Opening Searcher@69be616b[collection1] main
[junit4] 2> 885651 T6438 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 885652 T6438 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 885652 T6438 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 885652 T6438 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 885652 T6438 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 885653 T6438 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 885653 T6438 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 885653 T6438 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 885653 T6438 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 885653 T6438 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 885654 T6439 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69be616b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 885654 T6438 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 885655 T6442 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:43219 collection:collection1 shard:shard1
[junit4] 2> 885655 T6375 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 885656 T6375 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 885656 T6442 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 885663 T6442 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 885664 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 885664 T6442 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 885664 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1"} current state version: 4
[junit4] 2> 885664 T6442 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1007 name=collection1 org.apache.solr.core.SolrCore@3713496d url=https://127.0.0.1:43219/collection1 node=127.0.0.1:43219_ C1007_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=https://127.0.0.1:43219, node_name=127.0.0.1:43219_, state=down}
[junit4] 2> 885664 T6442 C1007 P43219 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:43219/collection1/
[junit4] 2> 885665 T6442 C1007 P43219 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 885665 T6442 C1007 P43219 oasc.SyncStrategy.syncToMe https://127.0.0.1:43219/collection1/ has no replicas
[junit4] 2> 885665 T6442 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:43219/collection1/ shard1
[junit4] 2> 885665 T6407 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
[junit4] 2> 885665 T6442 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 885665 T6437 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 885666 T6437 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2
[junit4] 2> 885668 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 885669 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"https://127.0.0.1:43219",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 885670 T6407 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 2
[junit4] 2> 885670 T6437 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 885671 T6437 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3
[junit4] 2> 885719 T6442 oasc.ZkController.register We are https://127.0.0.1:43219/collection1/ and leader is https://127.0.0.1:43219/collection1/
[junit4] 2> 885719 T6442 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:43219
[junit4] 2> 885719 T6442 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 885719 T6442 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 885720 T6442 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 885720 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 885721 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:43219",
[junit4] 2> "node_name":"127.0.0.1:43219_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"} current state version: 4
[junit4] 2> 885722 T6407 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:43219",
[junit4] 2> "node_name":"127.0.0.1:43219_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 885722 T6407 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 3
[junit4] 2> 885723 T6437 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 885723 T6442 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4
[junit4] 2> 885724 T6437 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4
[junit4] 2> 885899 T6375 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 885900 T6375 oejs.Server.doStart jetty-9.2.6.v20141205
[junit4] 2> 885901 T6375 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@3fbf9092{/,null,AVAILABLE}
[junit4] 2> 885904 T6375 oejs.AbstractConnector.doStart Started ServerConnector@5ddd5673{SSL-http/1.1}{127.0.0.1:59232}
[junit4] 2> 885904 T6375 oejs.Server.doStart Started @886932ms
[junit4] 2> 885906 T6375 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 885906 T6375 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 885906 T6375 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-004
[junit4] 2> 885907 T6375 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-004/'
[junit4] 2> 885921 T6375 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-004/solr.xml
[junit4] 2> 885929 T6375 oasc.CoreContainer.<init> New CoreContainer 1539605580
[junit4] 2> 885929 T6375 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-004/]
[junit4] 2> 885930 T6375 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 885930 T6375 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 885930 T6375 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 885930 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 885930 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 885931 T6375 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 885931 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 885931 T6375 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 885931 T6375 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 885931 T6375 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 885932 T6375 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 885933 T6375 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 885933 T6375 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 885934 T6375 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 885934 T6375 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54337/solr
[junit4] 2> 885934 T6375 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 885934 T6375 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 885935 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 885936 T6456 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a721be0 name:ZooKeeperConnection Watcher:127.0.0.1:54337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 885936 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 885937 T6375 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 885938 T6375 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 885939 T6459 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4eac343b name:ZooKeeperConnection Watcher:127.0.0.1:54337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 885939 T6375 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 885942 T6375 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 886945 T6375 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59232_
[junit4] 2> 886946 T6375 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59232_
[junit4] 2> 886948 T6375 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 886951 T6460 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 886952 T6460 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 886952 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 886953 T6460 oasc.ZkController.preRegister Registering watch for external collection collection1
[junit4] 2> 886953 T6460 oascc.ZkStateReader.addZkWatch addZkWatch collection1
[junit4] 2> 886953 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:59232",
[junit4] 2> "node_name":"127.0.0.1:59232_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"} current state version: 4
[junit4] 2> 886953 T6460 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper...
[junit4] 2> 886954 T6407 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:59232",
[junit4] 2> "node_name":"127.0.0.1:59232_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 886954 T6407 oasco.ReplicaMutator.updateState Collection already exists with numShards=1
[junit4] 2> 886954 T6460 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4
[junit4] 2> 886954 T6407 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 886954 T6460 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 886955 T6407 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 4
[junit4] 2> 886955 T6437 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 886955 T6459 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 886956 T6437 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5
[junit4] 2> 886958 T6459 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5
[junit4] 2> 887955 T6460 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 887955 T6460 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 887955 T6460 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 887955 T6460 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 887956 T6460 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 887956 T6460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-004/collection1/'
[junit4] 2> 887957 T6460 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest%20273BBFE4963E4923-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 887957 T6460 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest%20273BBFE4963E4923-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 887957 T6460 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest%20273BBFE4963E4923-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 887966 T6460 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
[junit4] 2> 887968 T6460 oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 887971 T6460 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 887984 T6460 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 887993 T6460 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 887994 T6460 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 887998 T6460 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 888087 T6460 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 888088 T6460 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 888089 T6460 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 888091 T6460 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 888101 T6460 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 888101 T6460 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 888102 T6460 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/
[junit4] 2> 888102 T6460 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@615e5c33
[junit4] 2> 888102 T6460 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2
[junit4] 2> 888103 T6460 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index/
[junit4] 2> 888103 T6460 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 888103 T6460 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index
[junit4] 2> 888104 T6460 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=4.6474609375, floorSegmentMB=2.1572265625, forceMergeDeletesPctAllowed=9.000088723067844, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7718117633583595
[junit4] 2> 888112 T6460 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 888112 T6460 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 888116 T6460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 888117 T6460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 888117 T6460 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 888117 T6460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 888117 T6460 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 888118 T6460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 888118 T6460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 888118 T6460 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 888118 T6460 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 888119 T6460 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 888119 T6460 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 888120 T6460 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 888120 T6460 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 888121 T6460 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
[junit4] 2> 888121 T6460 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
[junit4] 2> 888123 T6460 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
[junit4] 2> 888123 T6460 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
[junit4] 2> 888124 T6460 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
[junit4] 2> 888124 T6460 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
[junit4] 2> 888125 T6460 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
[junit4] 2> 888125 T6460 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
[junit4] 2> 888125 T6460 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
[junit4] 2> 888126 T6460 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
[junit4] 2> 888126 T6460 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
[junit4] 2> 888127 T6460 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
[junit4] 2> 888127 T6460 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 888128 T6460 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 888130 T6460 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 888131 T6460 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 888132 T6460 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 888132 T6460 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 888133 T6460 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 888134 T6460 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 888134 T6460 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 888136 T6460 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 888136 T6460 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 888137 T6460 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 888137 T6460 oass.SolrIndexSearcher.<init> Opening Searcher@51e873c7[collection1] main
[junit4] 2> 888137 T6460 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 888138 T6460 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 888138 T6460 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 888138 T6460 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 888139 T6460 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 888139 T6460 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 888139 T6460 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 888139 T6460 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 888140 T6460 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 888140 T6460 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 888141 T6461 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51e873c7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 888141 T6460 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 888142 T6464 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:59232 collection:collection1 shard:shard1
[junit4] 2> 888142 T6375 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 888142 T6375 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 888143 T6464 oasc.ZkController.register We are https://127.0.0.1:59232/collection1/ and leader is https://127.0.0.1:43219/collection1/
[junit4] 2> 888143 T6464 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:59232
[junit4] 2> 888144 T6464 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 888144 T6464 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C1008 name=collection1 org.apache.solr.core.SolrCore@7de5fc0e url=https://127.0.0.1:59232/collection1 node=127.0.0.1:59232_ C1008_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=https://127.0.0.1:59232, node_name=127.0.0.1:59232_, state=down}
[junit4] 2> 888144 T6465 C1008 P59232 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 888145 T6465 C1008 P59232 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 888146 T6465 C1008 P59232 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is https://127.0.0.1:43219/collection1/ and I am https://127.0.0.1:59232/collection1/
[junit4] 2> 888146 T6465 C1008 P59232 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 888146 T6465 C1008 P59232 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 888147 T6406 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 888148 T6407 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:59232",
[junit4] 2> "node_name":"127.0.0.1:59232_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"recovering",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"} current state version: 4
[junit4] 2> 888148 T6464 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5
[junit4] 2> 888148 T6407 oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:59232",
[junit4] 2> "node_name":"127.0.0.1:59232_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"recovering",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 888149 T6407 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 5
[junit4] 2> 888150 T6459 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 888150 T6437 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 888150 T6459 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6
[junit4] 2> 888151 T6437 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6
[junit4] 2> 888156 T6375 oasc.AbstractFullDistribZkTestBase.enableAutoSoftCommit Turning on auto soft commit: 1000
[junit4] 2> C1008_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=https://127.0.0.1:59232, node_name=127.0.0.1:59232_, state=recovering}
[junit4] 2> 888156 T6465 C1008 P59232 oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to https://127.0.0.1:43219; WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1%3A59232_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
[junit4] 2> ASYNC NEW_CORE C1009 name=collection1 org.apache.solr.core.SolrCore@25a03f08 url=https://127.0.0.1:39766/collection1 node=127.0.0.1:39766_ C1009_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:39766, node_name=127.0.0.1:39766_, state=active, leader=true}
[junit4] 2> 888218 T6396 C1009 P39766 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1490089029825200128)} 0 3
[junit4] 2> 888234 T6427 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 888237 T6427 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6
[junit4] 2> 888237 T6427 oasha.CoreAdminHandler.handleWaitForSt
[...truncated too long message...]
.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 8103585 T6375 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 8103585 T6375 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index;done=true>>]
[junit4] 2> 8103586 T6375 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index
[junit4] 2> 8103589 T6375 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index
[junit4] 2> 8103609 T6375 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 8103609 T6375 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index;done=false>>]
[junit4] 2> 8103609 T6375 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2/index
[junit4] 2> 8103610 T6375 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2;done=false>>]
[junit4] 2> 8103610 T6375 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001/tempDir-001/jetty2
[junit4] 2> 8104089 T6457 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 8104090 T6375 oejs.AbstractConnector.doStop Stopped ServerConnector@5ddd5673{SSL-http/1.1}{127.0.0.1:0}
[junit4] 2> 8104091 T6375 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@3fbf9092{/,null,UNAVAILABLE}
[junit4] 2> 8104092 T6375 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 8104092 T6375 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54337 54337
[junit4] 2> 8104787 T6375 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest 273BBFE4963E4923-001
[junit4] 2> 7223299 T6374 ccr.ThreadLeakControl.tryToInterruptAll All leaked threads terminated.
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_OM, timezone=America/Havana
[junit4] 2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=8,threads=1,free=152643504,total=375914496
[junit4] 2> NOTE: All tests run in this JVM: [TestReplicationHandler, TestDistributedSearch, TestCollectionAPI, RankQueryTest, JSONWriterTest, SyncSliceTest, BlockCacheTest, SolrIndexConfigTest, TestArbitraryIndexDir, TestBinaryResponseWriter, TestSort, TestSolrXmlPersistor, TestDistributedMissingSort, BJQParserTest, PluginInfoTest, DistributedDebugComponentTest, HighlighterConfigTest, MultiThreadedOCPTest, TestStressRecovery, NoCacheHeaderTest, OverseerTest, OutputWriterTest, SoftAutoCommitTest, SharedFSAutoReplicaFailoverUtilsTest, TestFuzzyAnalyzedSuggestions, TestInitQParser, BasicDistributedZkTest, SuggesterWFSTTest, TestStressReorder, TestSurroundQueryParser, TestManagedSchemaDynamicFieldResource, TestUpdate, TestFastWriter, TestRebalanceLeaders, TestFieldResource, MinimalSchemaTest, ResourceLoaderTest, TestSolrCoreProperties, TestSolrQueryParserResource, DirectUpdateHandlerTest, CoreAdminHandlerTest, TestBinaryField, ShowFileRequestHandlerTest, OverriddenZkACLAndCredentialsProvidersTest, UnloadDistributedZkTest, TestLeaderElectionZkExpiry, TestCSVLoader, HttpPartitionTest, HighlighterMaxOffsetTest, EnumFieldTest, CoreContainerCoreInitFailuresTest, TestReRankQParserPlugin, AnalysisAfterCoreReloadTest, TestClusterStateMutator, TestOverriddenPrefixQueryForCustomFieldType, SolrCoreCheckLockOnStartupTest, TestManagedResourceStorage, TestPartialUpdateDeduplication, TestLazyCores, RequiredFieldsTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestImplicitCoreProperties, SliceStateTest, TestQuerySenderListener, SimpleCollectionCreateDeleteTest, TestIBSimilarityFactory, PrimUtilsTest, CloudMLTQParserTest, TestHighFrequencyDictionaryFactory, FacetPivotSmallTest, CoreAdminRequestStatusTest, TestManagedSynonymFilterFactory, TestSolrDynamicMBean, TestCodecSupport, HdfsRecoveryZkTest, TestDocumentBuilder, PreAnalyzedFieldTest, TestSolrConfigHandler, TestRequestStatusCollectionAPI, TestStressLucene, DistributedQueryElevationComponentTest, TestSolrDeletionPolicy2, PathHierarchyTokenizerFactoryTest, MergeStrategyTest, SuggestComponentTest, DeleteInactiveReplicaTest, NumericFieldsTest, TestRandomDVFaceting, DefaultValueUpdateProcessorTest, TestUniqueKeyFieldResource, RemoteQueryErrorTest, LukeRequestHandlerTest, DistanceFunctionTest, HdfsSyncSliceTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestSolrXmlPersistence, TestAnalyzedSuggestions, TestCursorMarkWithoutUniqueKey, CursorMarkTest, CircularListTest, TestMaxScoreQueryParser, IndexSchemaRuntimeFieldTest, DOMUtilTest, RecoveryZkTest, TestDocBasedVersionConstraints, TestCoreContainer, SolrInfoMBeanTest, TestSweetSpotSimilarityFactory, TestQuerySenderNoQuery, UUIDFieldTest, TestNoOpRegenerator, TestLRUCache, ZkControllerTest, TestQueryUtils, ChaosMonkeySafeLeaderTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=273BBFE4963E4923 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_OM -Dtests.timezone=America/Havana -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J0 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([273BBFE4963E4923]:0)
[junit4] Completed on J0 in 7223.33s, 1 test, 2 errors <<< FAILURES!
[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 461 suites, 1870 tests, 1 suite-level error, 1 error, 47 ignored (22 assumptions)
Total time: 160 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any