You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2015/12/29 13:11:00 UTC

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 896 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/896/

1 tests failed.
FAILED:  org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny

Error Message:
maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode

Stack Trace:
java.lang.IllegalArgumentException: maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
	at __randomizedtesting.SeedInfo.seed([B6A7B02C094A1978:181EDD6FE19860C1]:0)
	at org.apache.lucene.util.bkd.BKDWriter.<init>(BKDWriter.java:161)
	at org.apache.lucene.codecs.lucene60.Lucene60DimensionalWriter.writeField(Lucene60DimensionalWriter.java:88)
	at org.apache.lucene.index.DimensionalValuesWriter.flush(DimensionalValuesWriter.java:68)
	at org.apache.lucene.index.DefaultIndexingChain.writeDimensionalValues(DefaultIndexingChain.java:146)
	at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:96)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:425)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3099)
	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3074)
	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1727)
	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1707)
	at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
	at org.apache.lucene.search.TestDimensionalRangeQuery.verifyBinary(TestDimensionalRangeQuery.java:508)
	at org.apache.lucene.search.TestDimensionalRangeQuery.doTestRandomBinary(TestDimensionalRangeQuery.java:419)
	at org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny(TestDimensionalRangeQuery.java:375)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 1700 lines...]
   [junit4] Suite: org.apache.lucene.search.TestDimensionalRangeQuery
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDimensionalRangeQuery -Dtests.method=testRandomBinaryTiny -Dtests.seed=B6A7B02C094A1978 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_BO -Dtests.timezone=SystemV/EST5 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.02s J0 | TestDimensionalRangeQuery.testRandomBinaryTiny <<<
   [junit4]    > Throwable #1: java.lang.IllegalArgumentException: maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B6A7B02C094A1978:181EDD6FE19860C1]:0)
   [junit4]    > 	at org.apache.lucene.util.bkd.BKDWriter.<init>(BKDWriter.java:161)
   [junit4]    > 	at org.apache.lucene.codecs.lucene60.Lucene60DimensionalWriter.writeField(Lucene60DimensionalWriter.java:88)
   [junit4]    > 	at org.apache.lucene.index.DimensionalValuesWriter.flush(DimensionalValuesWriter.java:68)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.writeDimensionalValues(DefaultIndexingChain.java:146)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:96)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:425)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3099)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3074)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1727)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1707)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
   [junit4]    > 	at org.apache.lucene.search.TestDimensionalRangeQuery.verifyBinary(TestDimensionalRangeQuery.java:508)
   [junit4]    > 	at org.apache.lucene.search.TestDimensionalRangeQuery.doTestRandomBinary(TestDimensionalRangeQuery.java:419)
   [junit4]    > 	at org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny(TestDimensionalRangeQuery.java:375)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/test/J0/temp/lucene.search.TestDimensionalRangeQuery_B6A7B02C094A1978-001
   [junit4]   2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, locale=es_BO, timezone=SystemV/EST5
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=1,free=187269408,total=429391872
   [junit4]   2> NOTE: All tests run in this JVM: [TestSegmentInfos, TestConjunctionDISI, TestLucene54DocValuesFormat, TestMultiDocValues, TestBlockPostingsFormat3, TestPagedBytes, TestSentinelIntSet, TestNumericRangeQuery64, TestTermScorer, TestNativeFSLockFactory, TestLucene50FieldInfoFormat, TestVirtualMethod, TestFixedBitSet, TestIndexWriterUnicode, TestRollingBuffer, TestInfoStream, TestOrdinalMap, TestTerms, TestDocInverterPerFieldErrorInfo, Test2BNumericDocValues, TestGrowableByteArrayDataOutput, TestMergedIterator, FuzzyTermOnShortTermsTest, TestSegmentReader, TestAttributeSource, TestSloppyMath, TestWeakIdentityMap, TestDeterminism, TestIndexWriterExceptions2, TestTimeLimitingCollector, TestIndexSearcher, TestByteBlockPool, TestParallelTermEnum, TestIndexWriterCommit, TestDoc, TestCustomNorms, TestMinimize, TestIndexWriterOnVMError, TestBooleanMinShouldMatch, TestUniqueTermCount, TestFilterDirectory, TestOperations, TestStringHelper, FiniteStringsIteratorTest, TestNoMergePolicy, TestIndexWriterMergePolicy, TestMaxPosition, TestSizeBoundedForceMerge, TestSpanBoostQuery, TestAutomatonQuery, TestSimilarity2, TestSloppyPhraseQuery2, TestParallelCompositeReader, TestPerFieldPostingsFormat, MultiCollectorTest, Test2BSortedDocValuesOrds, TestUnicodeUtil, TestDocValuesIndexing, TestIndexWriterLockRelease, TestMultiTermConstantScore, TestThreadedForceMerge, TestForTooMuchCloning, TestPositionIncrement, TestMutableValues, TestReusableStringReader, TestIndexWriterConfig, TestSearcherManager, TestHighCompressionMode, TestDocumentsWriterStallControl, TestExceedMaxTermLength, TestFilterDirectoryReader, TestIndexReaderClose, TestIndexWriterFromReader, TestIndexWriterThreadsToSegments, TestReaderWrapperDVTypeCheck, TestStressDeletes, TestApproximationSearchEquivalence, TestBooleanCoord, TestBooleanRewrites, TestControlledRealTimeReopenThread, TestDimensionalRangeQuery]
   [junit4] Completed [403/403 (1!)] on J0 in 757.26s, 18 tests, 1 error <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:50: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:991: There were test failures: 403 suites (1 ignored), 3317 tests, 1 error, 29 ignored (25 assumptions) [seed: B6A7B02C094A1978]

Total time: 61 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 897 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/897/

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [TransactionLog]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [TransactionLog]
	at __randomizedtesting.SeedInfo.seed([D77AACF2B2B5CCE7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
	at __randomizedtesting.SeedInfo.seed([D77AACF2B2B5CCE7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory]
	at __randomizedtesting.SeedInfo.seed([D77AACF2B2B5CCE7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10412 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/init-core-data-001
   [junit4]   2> 1341384 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[D77AACF2B2B5CCE7]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1341384 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[D77AACF2B2B5CCE7]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /a_y/
   [junit4]   2> 1341387 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1341387 INFO  (Thread-5363) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1341387 INFO  (Thread-5363) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1341487 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkTestServer start zk server on port:54397
   [junit4]   2> 1341487 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341488 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341490 INFO  (zkCallback-1523-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@298c9871 name:ZooKeeperConnection Watcher:127.0.0.1:54397 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341490 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341490 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341490 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1341492 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341494 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341496 INFO  (zkCallback-1524-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@40d20658 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341496 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341496 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341496 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1341500 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1341503 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1341505 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1341508 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1341508 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1341512 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1341512 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1341514 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1341514 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1341516 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1341516 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1341518 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1341518 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1341520 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1341520 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1341525 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1341525 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1341527 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1341527 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1341530 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1341530 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1341533 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1341533 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1341537 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1341537 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1341541 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341542 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341543 INFO  (zkCallback-1525-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4544abca name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341543 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341544 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341872 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1
   [junit4]   2> 1341874 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1341876 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@18f912e1{/a_y,null,AVAILABLE}
   [junit4]   2> 1341877 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.u.s.SslContextFactory x509=X509@2811708d(solrtest,h=[],w=[]) for SslContextFactory@24bac6db(file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore)
   [junit4]   2> 1341880 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.ServerConnector Started ServerConnector@780f7fe3{SSL,[ssl, http/1.1]}{127.0.0.1:41101}
   [junit4]   2> 1341880 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.Server Started @1344907ms
   [junit4]   2> 1341880 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/tempDir-001/control/data, hostContext=/a_y, hostPort=41101, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores}
   [junit4]   2> 1341880 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1341881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001'
   [junit4]   2> 1341881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1341881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1341881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341883 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341898 INFO  (zkCallback-1526-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2aecd07c name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341898 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341898 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341899 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1341899 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/solr.xml
   [junit4]   2> 1341909 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores
   [junit4]   2> 1341909 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer New CoreContainer 926607776
   [junit4]   2> 1341909 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001]
   [junit4]   2> 1341910 WARN  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/lib
   [junit4]   2> 1341911 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1341919 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1341921 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1341921 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1341922 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54397/solr
   [junit4]   2> 1341922 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1341922 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341923 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341926 INFO  (zkCallback-1529-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@10eca9e9 name:ZooKeeperConnection Watcher:127.0.0.1:54397 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341927 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341931 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341936 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341941 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7df0de35 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341941 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341943 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1341945 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1341949 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1341952 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1341955 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1341960 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1341961 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1341966 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1341967 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1341969 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41101_a_y
   [junit4]   2> 1341970 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:41101_a_y
   [junit4]   2> 1341972 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1341973 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1341975 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1341975 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95123646694686725-127.0.0.1:41101_a_y-n_0000000000
   [junit4]   2> 1341976 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41101_a_y
   [junit4]   2> 1341976 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1341977 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer Overseer (id=95123646694686725-127.0.0.1:41101_a_y-n_0000000000) starting
   [junit4]   2> 1341980 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1341991 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1341992 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1341993 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1341996 INFO  (OverseerCollectionConfigSetProcessor-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1341999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1342032 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1342032 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1342032 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores
   [junit4]   2> 1342033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1342033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1
   [junit4]   2> 1342033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1342035 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1342035 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1342036 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1342036 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1342037 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:41101_a_y",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1342038 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1342038 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:41101_a_y",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1342039 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1342039 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1342042 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1342043 INFO  (zkCallback-1530-thread-2-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343038 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1343038 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1343039 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1343039 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1343040 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1343040 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1'
   [junit4]   2> 1343040 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1343040 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1343041 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1343047 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1343051 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1343062 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1343069 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1343182 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1343183 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1343184 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1343187 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore solr.StandardDirectoryFactory
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/]
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30f1226b
   [junit4]   2> 1343211 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data
   [junit4]   2> 1343211 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1343212 WARN  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1343212 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index
   [junit4]   2> 1343212 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=69.69921875, floorSegmentMB=2.1015625, forceMergeDeletesPctAllowed=29.51221115478454, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1343222 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1343222 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1343223 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1343228 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1343229 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1343229 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1343229 WARN  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1343232 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1343233 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1343233 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1343234 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1343237 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1343238 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1343238 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1343238 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1343240 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 1343240 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1343240 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1343241 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1343241 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1343241 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@7698919f[collection1] main
   [junit4]   2> 1343241 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1343245 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1343245 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1343249 INFO  (searcherExecutor-2511-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7698919f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1521978349065338880
   [junit4]   2> 1343252 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 3.0ms to seed version buckets with highest version 1521978349065338880
   [junit4]   2> 1343252 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1343253 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1343254 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:https://127.0.0.1:41101/a_y collection:control_collection shard:shard1
   [junit4]   2> 1343255 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1343262 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95123646694686725-core_node1-n_0000000000
   [junit4]   2> 1343262 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1343264 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:41101/a_y/collection1/
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:41101/a_y/collection1/ has no replicas
   [junit4]   2> 1343266 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1343271 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 1343274 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:41101/a_y/collection1/ shard1
   [junit4]   2> 1343274 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1343378 INFO  (zkCallback-1530-thread-2-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343378 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are https://127.0.0.1:41101/a_y/collection1/ and leader is https://127.0.0.1:41101/a_y/collection1/
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:41101/a_y
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1343429 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:41101_a_y",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1343429 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:41101_a_y",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1343532 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343532 INFO  (zkCallback-1530-thread-2-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343540 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1343559 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1343571 INFO  (zkCallback-1532-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6c0cd8cd name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1343572 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1343572 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1343572 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1343574 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1343574 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1343574 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1343587 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1343596 INFO  (zkCallback-1533-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@54954eb4 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1343596 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1343596 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1343597 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"4",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 1343598 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1343700 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 1343953 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores/collection1
   [junit4]   2> 1343954 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001
   [junit4]   2> 1343955 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1343957 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@58785e2a{/a_y,null,AVAILABLE}
   [junit4]   2> 1343958 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.u.s.SslContextFactory x509=X509@25c57cf0(solrtest,h=[],w=[]) for SslContextFactory@1a05ce2c(file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore)
   [junit4]   2> 1343977 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.ServerConnector Started ServerConnector@760b2cd8{SSL,[ssl, http/1.1]}{127.0.0.1:34831}
   [junit4]   2> 1343980 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.Server Started @1347007ms
   [junit4]   2> 1343980 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/a_y, hostPort=34831, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores}
   [junit4]   2> 1343980 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1343981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001'
   [junit4]   2> 1343981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1343981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1343981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1343983 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1343985 INFO  (zkCallback-1534-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@43114d97 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1343985 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1343985 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1343986 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1343986 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/solr.xml
   [junit4]   2> 1343991 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores
   [junit4]   2> 1343991 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer New CoreContainer 2085268738
   [junit4]   2> 1343991 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001]
   [junit4]   2> 1343991 WARN  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/lib
   [junit4]   2> 1343995 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1343997 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54397/solr
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1344001 INFO  (zkCallback-1537-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@344336b5 name:ZooKeeperConnection Watcher:127.0.0.1:54397 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1344002 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1344002 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1344004 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1344005 INFO  (zkCallback-1538-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7bc93874 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1344005 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1344012 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1345018 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34831_a_y
   [junit4]   2> 1345019 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:34831_a_y
   [junit4]   2> 1345020 INFO  (zkCallback-1532-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1345020 INFO  (zkCallback-1538-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1345020 INFO  (zkCallback-1530-thread-2-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1345020 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1345022 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1345022 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95123646694686730-127.0.0.1:34831_a_y-n_0000000001
   [junit4]   2> 1345023 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95123646694686725-127.0.0.1:41101_a_y-n_0000000000 to know if I could be the leader
   [junit4]   2> 1345033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1345033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1345034 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores
   [junit4]   2> 1345036 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1345036 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores/collection1
   [junit4]   2> 1345036 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1345054 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1345054 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1345055 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:34831/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:34831_a_y",
   [junit4]   2>   "numShards":"4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 1345055 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController Registering watch for collection collection1
   [junit4]   2> 1345056 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1]
   [junit4]   2> 1345056 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Add data for [collection1] ver [0]
   [junit4]   2> 1345056 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1345056 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Update state numShards=4 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:34831/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:34831_a_y",
   [junit4]   2>   "numShards":"4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1345056 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=4
   [junit4]   2> 1345056 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1345058 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1345058 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1345168 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 1345168 INFO  (zkCallback-1538-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1345169 INFO  (zkCallback-1538-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ZkStateReader Updating data for [collection1] from [0] to [1]
   [junit4]   2> 1346058 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1346058 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController Check for collection zkNode:collection1
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/collection1]
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/collection1] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores/collection1'
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1

[...truncated too long message...]

lr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:226)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   [junit4]   2> 	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:45)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:517)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> Dec 30, 2015 11:12:55 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {rnd_b=BlockTreeOrds(blocksize=128), _version_=PostingsFormat(name=Memory doPackFST= false), a_t=PostingsFormat(name=Memory doPackFST= false), a_i=BlockTreeOrds(blocksize=128), id=BlockTreeOrds(blocksize=128)}, docValues:{}, sim=ClassicSimilarity, locale=es_MX, timezone=Asia/Beirut
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=2,free=314799256,total=515899392
   [junit4]   2> NOTE: All tests run in this JVM: [ForceLeaderTest, FullHLLTest, TestUtils, TestSweetSpotSimilarityFactory, TestDocBasedVersionConstraints, AnalysisErrorHandlingTest, TestRTimerTree, HardAutoCommitTest, TestJmxIntegration, UUIDUpdateProcessorFallbackTest, BlockJoinFacetSimpleTest, TestSolrIndexConfig, CloneFieldUpdateProcessorFactoryTest, TestDefaultStatsCache, TestShortCircuitedRequests, TestScoreJoinQPScore, AnalyticsMergeStrategyTest, TestCSVResponseWriter, DebugComponentTest, OverseerTest, TestComplexPhraseQParserPlugin, TestIndexSearcher, CoreAdminRequestStatusTest, HdfsChaosMonkeyNothingIsSafeTest, TestXIncludeConfig, XmlUpdateRequestHandlerTest, PrimitiveFieldTypeTest, DocValuesMultiTest, TestConfigSetsAPI, SuggestComponentTest, SolrTestCaseJ4Test, HdfsRestartWhileUpdatingTest, TestStressUserVersions, PluginInfoTest, TestInfoStreamLogging, ExitableDirectoryReaderTest, BadIndexSchemaTest, TestMiniSolrCloudClusterSSL, DirectUpdateHandlerTest, OverseerCollectionConfigSetProcessorTest, SmileWriterTest, TestExtendedDismaxParser, TestFieldTypeResource, CollectionTooManyReplicasTest, TestConfigSets, ScriptEngineTest, TestSolrJ, LukeRequestHandlerTest, BadCopyFieldTest, MergeStrategyTest, TestReloadDeadlock, TestCollationField, TestBulkSchemaConcurrent, TestRuleBasedAuthorizationPlugin, QueryParsingTest, TestBulkSchemaAPI, TestBlobHandler, DistributedExpandComponentTest, RequestHandlersTest, NoCacheHeaderTest, SolrCloudExampleTest, TestRawTransformer, VMParamsZkACLAndCredentialsProvidersTest, TestReqParamsAPI, TestUniqueKeyFieldResource, FieldMutatingUpdateProcessorTest, UnloadDistributedZkTest, TestClusterStateMutator, TestFieldCollectionResource, DistributedFacetPivotSmallAdvancedTest, SecurityConfHandlerTest, TestPHPSerializedResponseWriter, TestGroupingSearch, TestMaxScoreQueryParser, BlockCacheTest, BasicDistributedZk2Test, SolrCoreCheckLockOnStartupTest, ParsingFieldUpdateProcessorsTest, TestSchemaManager, DistribJoinFromCollectionTest, TestSerializedLuceneMatchVersion, TestExactStatsCache, DocValuesMissingTest, TestWriterPerf, TestOverriddenPrefixQueryForCustomFieldType, TestSolrQueryParser, DistributedQueryComponentCustomSortTest, BadComponentTest, TestDefaultSearchFieldResource, CircularListTest, DirectUpdateHandlerOptimizeTest, HighlighterMaxOffsetTest, TestFilteredDocIdSet, TestStressRecovery, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, LeaderElectionIntegrationTest, TestReplicationHandler, DistributedTermsComponentTest, TestSolr4Spatial, StatsComponentTest, BasicFunctionalityTest, TestLazyCores, TestBadConfig, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SuggesterFSTTest, CoreAdminHandlerTest, TestStressLucene, SpatialFilterTest, WordBreakSolrSpellCheckerTest, TestWordDelimiterFilterFactory, TestSolrDeletionPolicy1, SolrInfoMBeanTest, CacheHeaderTest, TestQueryUtils, TestReversedWildcardFilterFactory, TestOmitPositions, TermsComponentTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, MoreLikeThisHandlerTest, TestArbitraryIndexDir, RegexBoostProcessorTest, SolrPluginUtilsTest, UniqFieldsUpdateProcessorFactoryTest, PingRequestHandlerTest, DirectSolrConnectionTest, SpellPossibilityIteratorTest, TestCharFilters, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TestJmxMonitoredMap, URLClassifyProcessorTest, DateMathParserTest, TestSystemIdResolver, TestSuggestSpellingConverter, SpellingQueryConverterTest, DOMUtilTest, ClusterStateTest, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, DistributedIntervalFacetingTest, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestTolerantSearch, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, ActionThrottleTest, AssignTest, AsyncCallRequestStatusResponseTest, CdcrReplicationDistributedZkTest, ReplicationFactorTest, SSLMigrationTest, SaslZkACLProviderTest, SimpleCollectionCreateDeleteTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudInspectUtil, TestCloudPivotFacet, TestDistribDocBasedVersion, TestMiniSolrCloudCluster, HdfsChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsChaosMonkeySafeLeaderTest -Dtests.seed=D77AACF2B2B5CCE7 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_MX -Dtests.timezone=Asia/Beirut -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | HdfsChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D77AACF2B2B5CCE7]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [483/573 (3!)] on J2 in 118.03s, 1 test, 1 failure <<< FAILURES!

[...truncated 301 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1424: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:981: There were test failures: 573 suites (8 ignored), 2284 tests, 3 suite-level errors, 33 ignored (18 assumptions) [seed: D77AACF2B2B5CCE7]

Total time: 205 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 896 - Still Failing

Posted by Michael McCandless <lu...@mikemccandless.com>.
I'll dig, silly test bug!

Mike McCandless

http://blog.mikemccandless.com


On Tue, Dec 29, 2015 at 4:11 AM, Apache Jenkins Server
<je...@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/896/
>
> 1 tests failed.
> FAILED:  org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny
>
> Error Message:
> maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
>
> Stack Trace:
> java.lang.IllegalArgumentException: maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
>         at __randomizedtesting.SeedInfo.seed([B6A7B02C094A1978:181EDD6FE19860C1]:0)
>         at org.apache.lucene.util.bkd.BKDWriter.<init>(BKDWriter.java:161)
>         at org.apache.lucene.codecs.lucene60.Lucene60DimensionalWriter.writeField(Lucene60DimensionalWriter.java:88)
>         at org.apache.lucene.index.DimensionalValuesWriter.flush(DimensionalValuesWriter.java:68)
>         at org.apache.lucene.index.DefaultIndexingChain.writeDimensionalValues(DefaultIndexingChain.java:146)
>         at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:96)
>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:425)
>         at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
>         at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
>         at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3099)
>         at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3074)
>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1727)
>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1707)
>         at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
>         at org.apache.lucene.search.TestDimensionalRangeQuery.verifyBinary(TestDimensionalRangeQuery.java:508)
>         at org.apache.lucene.search.TestDimensionalRangeQuery.doTestRandomBinary(TestDimensionalRangeQuery.java:419)
>         at org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny(TestDimensionalRangeQuery.java:375)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 1700 lines...]
>    [junit4] Suite: org.apache.lucene.search.TestDimensionalRangeQuery
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDimensionalRangeQuery -Dtests.method=testRandomBinaryTiny -Dtests.seed=B6A7B02C094A1978 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_BO -Dtests.timezone=SystemV/EST5 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR   0.02s J0 | TestDimensionalRangeQuery.testRandomBinaryTiny <<<
>    [junit4]    > Throwable #1: java.lang.IllegalArgumentException: maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([B6A7B02C094A1978:181EDD6FE19860C1]:0)
>    [junit4]    >        at org.apache.lucene.util.bkd.BKDWriter.<init>(BKDWriter.java:161)
>    [junit4]    >        at org.apache.lucene.codecs.lucene60.Lucene60DimensionalWriter.writeField(Lucene60DimensionalWriter.java:88)
>    [junit4]    >        at org.apache.lucene.index.DimensionalValuesWriter.flush(DimensionalValuesWriter.java:68)
>    [junit4]    >        at org.apache.lucene.index.DefaultIndexingChain.writeDimensionalValues(DefaultIndexingChain.java:146)
>    [junit4]    >        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:96)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:425)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3099)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3074)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1727)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1707)
>    [junit4]    >        at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
>    [junit4]    >        at org.apache.lucene.search.TestDimensionalRangeQuery.verifyBinary(TestDimensionalRangeQuery.java:508)
>    [junit4]    >        at org.apache.lucene.search.TestDimensionalRangeQuery.doTestRandomBinary(TestDimensionalRangeQuery.java:419)
>    [junit4]    >        at org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny(TestDimensionalRangeQuery.java:375)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/test/J0/temp/lucene.search.TestDimensionalRangeQuery_B6A7B02C094A1978-001
>    [junit4]   2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, locale=es_BO, timezone=SystemV/EST5
>    [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=1,free=187269408,total=429391872
>    [junit4]   2> NOTE: All tests run in this JVM: [TestSegmentInfos, TestConjunctionDISI, TestLucene54DocValuesFormat, TestMultiDocValues, TestBlockPostingsFormat3, TestPagedBytes, TestSentinelIntSet, TestNumericRangeQuery64, TestTermScorer, TestNativeFSLockFactory, TestLucene50FieldInfoFormat, TestVirtualMethod, TestFixedBitSet, TestIndexWriterUnicode, TestRollingBuffer, TestInfoStream, TestOrdinalMap, TestTerms, TestDocInverterPerFieldErrorInfo, Test2BNumericDocValues, TestGrowableByteArrayDataOutput, TestMergedIterator, FuzzyTermOnShortTermsTest, TestSegmentReader, TestAttributeSource, TestSloppyMath, TestWeakIdentityMap, TestDeterminism, TestIndexWriterExceptions2, TestTimeLimitingCollector, TestIndexSearcher, TestByteBlockPool, TestParallelTermEnum, TestIndexWriterCommit, TestDoc, TestCustomNorms, TestMinimize, TestIndexWriterOnVMError, TestBooleanMinShouldMatch, TestUniqueTermCount, TestFilterDirectory, TestOperations, TestStringHelper, FiniteStringsIteratorTest, TestNoMergePolicy, TestIndexWriterMergePolicy, TestMaxPosition, TestSizeBoundedForceMerge, TestSpanBoostQuery, TestAutomatonQuery, TestSimilarity2, TestSloppyPhraseQuery2, TestParallelCompositeReader, TestPerFieldPostingsFormat, MultiCollectorTest, Test2BSortedDocValuesOrds, TestUnicodeUtil, TestDocValuesIndexing, TestIndexWriterLockRelease, TestMultiTermConstantScore, TestThreadedForceMerge, TestForTooMuchCloning, TestPositionIncrement, TestMutableValues, TestReusableStringReader, TestIndexWriterConfig, TestSearcherManager, TestHighCompressionMode, TestDocumentsWriterStallControl, TestExceedMaxTermLength, TestFilterDirectoryReader, TestIndexReaderClose, TestIndexWriterFromReader, TestIndexWriterThreadsToSegments, TestReaderWrapperDVTypeCheck, TestStressDeletes, TestApproximationSearchEquivalence, TestBooleanCoord, TestBooleanRewrites, TestControlledRealTimeReopenThread, TestDimensionalRangeQuery]
>    [junit4] Completed [403/403 (1!)] on J0 in 757.26s, 18 tests, 1 error <<< FAILURES!
>
> [...truncated 11 lines...]
> BUILD FAILED
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:50: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1436: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:991: There were test failures: 403 suites (1 ignored), 3317 tests, 1 error, 29 ignored (25 assumptions) [seed: B6A7B02C094A1978]
>
> Total time: 61 minutes 45 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> No prior successful build to compare, so performing full copy of artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org