You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2015/08/15 11:31:57 UTC

[JENKINS] Lucene-Solr-NightlyTests-5.3 - Build # 6 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.3/6/

2 tests failed.
REGRESSION:  org.apache.lucene.search.TestGeoPointQuery.testRandomBig

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([656B7773F24A54BB]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestGeoPointQuery

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([656B7773F24A54BB]:0)




Build Log:
[...truncated 8882 lines...]
   [junit4] Suite: org.apache.lucene.search.TestGeoPointQuery
   [junit4]   2> Aug 15, 2015 6:31:33 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.search.TestGeoPointQuery
   [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1281)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1355)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:578)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:444)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:199)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:310)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
   [junit4]   2>    2) Thread[id=40, name=TEST-TestGeoPointQuery.testRandomBig-seed#[656B7773F24A54BB], state=TIMED_WAITING, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.lucene.store.SlowOpeningMockIndexInputWrapper.<init>(SlowOpeningMockIndexInputWrapper.java:34)
   [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:663)
   [junit4]   2>         at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.<init>(BlockTreeTermsReader.java:173)
   [junit4]   2>         at org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:446)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
   [junit4]   2>         at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)
   [junit4]   2>         at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
   [junit4]   2>         at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]   2>         at org.apache.lucene.index.BufferedUpdatesStream$SegmentState.<init>(BufferedUpdatesStream.java:384)
   [junit4]   2>         at org.apache.lucene.index.BufferedUpdatesStream.openSegmentStates(BufferedUpdatesStream.java:416)
   [junit4]   2>         at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:261)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3846)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3804)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
   [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1929)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4731)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:695)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4757)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4748)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1476)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1254)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.verify(TestGeoPointQuery.java:364)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.doTestRandom(TestGeoPointQuery.java:337)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.testRandomBig(TestGeoPointQuery.java:268)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    3) Thread[id=39, name=SUITE-TestGeoPointQuery-seed#[656B7773F24A54BB], state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1589)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:690)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:679)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$900(ThreadLeakControl.java:62)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:412)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:651)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:568)
   [junit4]   2>    4) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeoPointQuery -Dtests.method=testRandomBig -Dtests.seed=656B7773F24A54BB -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=lv_LV -Dtests.timezone=America/Argentina/Salta -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   7200s J0 | TestGeoPointQuery.testRandomBig <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([656B7773F24A54BB]:0)
   [junit4]   2> Aug 15, 2015 6:31:33 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Aug 15, 2015 6:31:53 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.search.TestGeoPointQuery: 
   [junit4]   2>    1) Thread[id=40, name=TEST-TestGeoPointQuery.testRandomBig-seed#[656B7773F24A54BB], state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.util.packed.MonotonicLongValues.get(MonotonicLongValues.java:42)
   [junit4]   2>         at org.apache.lucene.util.packed.PackedLongValues.get(PackedLongValues.java:112)
   [junit4]   2>         at org.apache.lucene.util.LongValues.get(LongValues.java:45)
   [junit4]   2>         at org.apache.lucene.index.MergeState$DocMap$1.get(MergeState.java:203)
   [junit4]   2>         at org.apache.lucene.index.MappingMultiPostingsEnum.nextDoc(MappingMultiPostingsEnum.java:104)
   [junit4]   2>         at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:128)
   [junit4]   2>         at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:1041)
   [junit4]   2>         at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:456)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
   [junit4]   2>         at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4089)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3664)
   [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1929)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4731)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:695)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4757)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4748)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1476)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1254)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.verify(TestGeoPointQuery.java:364)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.doTestRandom(TestGeoPointQuery.java:337)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.testRandomBig(TestGeoPointQuery.java:268)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Aug 15, 2015 6:31:53 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=40, name=TEST-TestGeoPointQuery.testRandomBig-seed#[656B7773F24A54BB], state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2> Aug 15, 2015 6:31:56 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=40, name=TEST-TestGeoPointQuery.testGeoDistanceQueryHuge-seed#[656B7773F24A54BB], state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.lang.StrictMath.sin(Native Method)
   [junit4]   2>         at org.apache.lucene.util.GeoProjectionUtils.llaToECF(GeoProjectionUtils.java:140)
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.lineCrossesSphere(GeoUtils.java:354)
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectCrossesCircle(GeoUtils.java:319)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl$GeoPointRadiusTermsEnum.cellCrosses(GeoPointDistanceQueryImpl.java:58)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl$GeoPointRadiusTermsEnum.cellIntersectsShape(GeoPointDistanceQueryImpl.java:68)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:104)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:82)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:107)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl$GeoPointRadiusTermsEnum.<init>(GeoPointDistanceQueryImpl.java:53)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl.getTermsEnum(GeoPointDistanceQueryImpl.java:42)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:323)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:769)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:695)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:411)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:440)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.geoDistanceQuery(TestGeoPointQuery.java:142)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.testGeoDistanceQueryHuge(TestGeoPointQuery.java:230)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Lucene53, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=lv_LV, timezone=America/Argentina/Salta
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=2,free=130240632,total=531628032
   [junit4]   2> NOTE: All tests run in this JVM: [TestDocValuesNumbersQuery, TestGeoPointQuery]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeoPointQuery -Dtests.seed=656B7773F24A54BB -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=lv_LV -Dtests.timezone=America/Argentina/Salta -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestGeoPointQuery (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([656B7773F24A54BB]:0)
   [junit4] Completed [17/19] on J0 in 7223.44s, 3 tests, 2 errors <<< FAILURES!

[...truncated 18 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:733: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/build.xml:470: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:2245: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/module-build.xml:58: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1449: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1003: There were test failures: 19 suites, 114 tests, 1 suite-level error, 1 error, 12 ignored

Total time: 244 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.3 - Build # 12 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.3/12/

3 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1440187414141,generation=2,filelist=[_6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Memory_0.ram, _6z.cfe, _6z.cfs, _6z.si, _70.cfe, _70.cfs, _70.si, _71.cfe, _71.cfs, _71.si, _72.cfe, _72.cfs, _72.si, _73.cfe, _73.cfs, _73.si, segments_2]}]> but was:<[{indexVersion=1440187414141,generation=3,filelist=[_6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Memory_0.ram, _74.cfe, _74.cfs, _74.si, segments_3]}, {indexVersion=1440187414141,generation=2,filelist=[_6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Memory_0.ram, _6z.cfe, _6z.cfs, _6z.si, _70.cfe, _70.cfs, _70.si, _71.cfe, _71.cfs, _71.si, _72.cfe, _72.cfs, _72.si, _73.cfe, _73.cfs, _73.si, segments_2]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1440187414141,generation=2,filelist=[_6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Memory_0.ram, _6z.cfe, _6z.cfs, _6z.si, _70.cfe, _70.cfs, _70.si, _71.cfe, _71.cfs, _71.si, _72.cfe, _72.cfs, _72.si, _73.cfe, _73.cfs, _73.si, segments_2]}]> but was:<[{indexVersion=1440187414141,generation=3,filelist=[_6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Memory_0.ram, _74.cfe, _74.cfs, _74.si, segments_3]}, {indexVersion=1440187414141,generation=2,filelist=[_6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Memory_0.ram, _6z.cfe, _6z.cfs, _6z.si, _70.cfe, _70.cfs, _70.si, _71.cfe, _71.cfs, _71.si, _72.cfe, _72.cfs, _72.si, _73.cfe, _73.cfs, _73.si, segments_2]}]>
	at __randomizedtesting.SeedInfo.seed([ABBC68D25DC3779C:8E6B73E22D8B799F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1138)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:51984: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:51984: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:301)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:418)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:168)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=22224, name=collection1, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=22224, name=collection1, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:55655: Could not find collection : awholynewstresscollection_collection1_0
	at __randomizedtesting.SeedInfo.seed([ABBC68D25DC3779C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)




Build Log:
[...truncated 10754 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/init-core-data-001
   [junit4]   2> 725246 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[ABBC68D25DC3779C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 725246 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[ABBC68D25DC3779C]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 725251 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 725253 INFO  (Thread-1952) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 725254 INFO  (Thread-1952) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 725353 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.ZkTestServer start zk server on port:33903
   [junit4]   2> 725354 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 725354 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 725357 INFO  (zkCallback-311-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@57b0ae60 name:ZooKeeperConnection Watcher:127.0.0.1:33903 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 725357 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 725357 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 725357 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 725360 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 725366 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 725366 INFO  (zkCallback-312-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@38ca0ebc name:ZooKeeperConnection Watcher:127.0.0.1:33903/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 725366 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 725367 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 725367 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 725368 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 725370 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 725371 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 725373 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 725373 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 725375 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 725376 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 725377 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 725378 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 725379 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 725379 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 725381 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 725381 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 725383 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 725383 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 725384 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 725385 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 725386 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 725386 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 725388 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 725388 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 725390 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 725390 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 725392 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 725392 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 725684 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1
   [junit4]   2> 725686 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 725687 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7dc5e23d{/,null,AVAILABLE}
   [junit4]   2> 725689 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.e.j.s.ServerConnector Started ServerConnector@3c150a75{HTTP/1.1}{127.0.0.1:40503}
   [junit4]   2> 725689 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.e.j.s.Server Started @728831ms
   [junit4]   2> 725689 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/tempDir-001/control/data, hostPort=40503, hostContext=/}
   [junit4]   2> 725689 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6b482747
   [junit4]   2> 725690 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/'
   [junit4]   2> 725706 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 725707 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 725708 INFO  (zkCallback-313-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@24fc02e0 name:ZooKeeperConnection Watcher:127.0.0.1:33903/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 725708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 725709 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 725710 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 725710 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/solr.xml
   [junit4]   2> 725725 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores
   [junit4]   2> 725725 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.CoreContainer New CoreContainer 269437805
   [junit4]   2> 725725 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/]
   [junit4]   2> 725725 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/lib
   [junit4]   2> 725725 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/lib).
   [junit4]   2> 725734 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 725735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 725735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 725736 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 725736 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33903/solr
   [junit4]   2> 725736 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 725736 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 725737 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 725738 INFO  (zkCallback-315-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3804f0d3 name:ZooKeeperConnection Watcher:127.0.0.1:33903 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 725739 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 725739 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 725741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 725742 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1554e733 name:ZooKeeperConnection Watcher:127.0.0.1:33903/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 725742 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 725743 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 725745 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 725747 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 725750 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 725752 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 725755 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 725757 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 725758 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 725760 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 725766 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40503_
   [junit4]   2> 725766 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:40503_
   [junit4]   2> 725768 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 725769 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 725771 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 725771 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94384023632936964-127.0.0.1:40503_-n_0000000000
   [junit4]   2> 725772 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40503_
   [junit4]   2> 725772 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 725774 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.Overseer Overseer (id=94384023632936964-127.0.0.1:40503_-n_0000000000) starting
   [junit4]   2> 725775 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 725782 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 725783 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 725786 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 725786 INFO  (OverseerCollectionProcessor-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.OverseerCollectionProcessor Process current queue of collection creations
   [junit4]   2> 725808 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 725808 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 725809 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores
   [junit4]   2> 725810 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.SolrCore Created CoreDescriptor: {transient=false, collection=control_collection, coreNodeName=, shard=, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1, dataDir=data/, loadOnStartup=true, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/, name=collection1, configSetProperties=configsetprops.json, schema=schema.xml, config=solrconfig.xml}
   [junit4]   2> 725810 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/
   [junit4]   2> 725810 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 725811 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 725811 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 725812 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1
   [junit4]   2> 725812 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:40503_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 725812 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 725812 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 725813 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://127.0.0.1:40503",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "node_name":"127.0.0.1:40503_",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "collection":"control_collection"} current state version: 0
   [junit4]   2> 725814 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://127.0.0.1:40503",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "node_name":"127.0.0.1:40503_",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "collection":"control_collection"}
   [junit4]   2> 725814 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 725814 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 725815 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 725817 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 1
   [junit4]   2> 726812 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 726813 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 726813 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 726813 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 726814 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 726814 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/'
   [junit4]   2> 726830 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 726839 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 726853 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.3.0
   [junit4]   2> 726883 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 726884 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 726892 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 727154 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 727163 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 727165 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 727175 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 727179 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 727182 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 727182 WARN  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 727183 WARN  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 727183 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 727183 WARN  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 727183 WARN  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 727184 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1
   [junit4]   2> 727185 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 727185 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 727185 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 727185 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ffe9354
   [junit4]   2> 727186 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/data
   [junit4]   2> 727186 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/data/index/
   [junit4]   2> 727186 WARN  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 727186 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/data/index
   [junit4]   2> 727187 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=24, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.19899943985760293]
   [junit4]   2> 727187 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@6c363714 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5b5a398c),segFN=segments_1,generation=1}
   [junit4]   2> 727187 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 727192 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 727192 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 727193 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 727193 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 727193 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 727193 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 727193 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 727194 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 727194 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 727197 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 727198 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/data/
   [junit4]   2> 727198 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 727198 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 727201 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 727202 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 727206 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: standard,/update,/admin/ping,/admin/plugins,/get,/admin/segments,/admin/file,/config,/admin/properties,/admin/luke,/update/csv,/admin/threads,/replication,/update/json/docs,/admin/logging,/admin/mbeans,/schema,/admin/system,/update/json
   [junit4]   2> 727207 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 727208 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 727208 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 727209 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 727209 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 727209 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=88.6494140625, floorSegmentMB=1.1689453125, forceMergeDeletesPctAllowed=17.60987502812817, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 727210 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@6c363714 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5b5a398c),segFN=segments_1,generation=1}
   [junit4]   2> 727210 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 727210 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@2bd27424[collection1] main
   [junit4]   2> 727210 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 727212 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 727212 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 727212 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 727212 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 727212 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 727213 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 727213 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 727213 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 727213 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 727214 INFO  (searcherExecutor-1104-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2bd27424[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 727214 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 727214 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 727215 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 727215 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1510144380182200320
   [junit4]   2> 727217 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 3 ms to seed version buckets with highest version 1510144380182200320
   [junit4]   2> 727217 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 727218 INFO  (coreLoadExecutor-1103-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 727218 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:40503 collection:control_collection shard:shard1
   [junit4]   2> 727219 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 727223 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94384023632936964-core_node1-n_0000000000
   [junit4]   2> 727224 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 727225 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 727226 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 727226 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 727226 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 727227 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40503/collection1/
   [junit4]   2> 727227 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 727227 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40503/collection1/ has no replicas
   [junit4]   2> 727227 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40503/collection1/ shard1
   [junit4]   2> 727228 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 727236 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:40503",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 727238 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 727239 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 727240 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 2
   [junit4]   2> 727282 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:40503/collection1/ and leader is http://127.0.0.1:40503/collection1/
   [junit4]   2> 727282 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40503
   [junit4]   2> 727282 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 727282 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 727282 INFO  (coreZkRegister-1097-thread-1-processing-r:core_node1 n:127.0.0.1:40503_ s:shard1 x:collection1 c:control_collection) [n:127.0.0.1:40503_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 727284 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 727285 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://127.0.0.1:40503",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"127.0.0.1:40503_",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "collection":"control_collection"} current state version: 2
   [junit4]   2> 727286 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://127.0.0.1:40503",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "node_name":"127.0.0.1:40503_",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "collection":"control_collection"}
   [junit4]   2> 727314 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 727316 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 727317 INFO  (zkCallback-318-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@13c6ea91 name:ZooKeeperConnection Watcher:127.0.0.1:33903/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727317 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 727317 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 727318 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 727320 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 727321 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 727321 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 727322 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 727323 INFO  (zkCallback-319-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c27091d name:ZooKeeperConnection Watcher:127.0.0.1:33903/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727323 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 727323 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 727325 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 727326 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 2
   [junit4]   2> 727326 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 727327 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 727327 INFO  (zkCallback-318-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 727329 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 727337 INFO  (zkCallback-318-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 727432 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 727433 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 727433 INFO  (zkCallback-318-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 727435 INFO  (zkCallback-318-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 727435 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 727789 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/cores/collection1
   [junit4]   2> 727791 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001
   [junit4]   2> 727800 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 727869 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7e21d032{/,null,AVAILABLE}
   [junit4]   2> 727869 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.e.j.s.ServerConnector Started ServerConnector@2163a7b5{HTTP/1.1}{127.0.0.1:51984}
   [junit4]   2> 727869 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.e.j.s.Server Started @731011ms
   [junit4]   2> 727870 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=51984, solrconfig=solrconfig.xml, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/cores, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/tempDir-001/jetty1}
   [junit4]   2> 727870 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6b482747
   [junit4]   2> 727870 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/'
   [junit4]   2> 727902 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 727904 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 727905 INFO  (zkCallback-320-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@601e62a9 name:ZooKeeperConnection Watcher:127.0.0.1:33903/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727905 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 727905 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 727906 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 727906 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/solr.xml
   [junit4]   2> 727921 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/cores
   [junit4]   2> 727922 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.CoreContainer New CoreContainer 385831866
   [junit4]   2> 727922 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/]
   [junit4]   2> 727922 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/lib
   [junit4]   2> 727922 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/lib).
   [junit4]   2> 727931 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 727932 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 727932 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 727932 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 727933 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33903/solr
   [junit4]   2> 727933 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 727933 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 727943 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 727944 INFO  (zkCallback-322-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2d5608cd name:ZooKeeperConnection Watcher:127.0.0.1:33903 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727945 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 727946 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 727951 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 727954 INFO  (zkCallback-323-thread-1-processing-n:127.0.0.1:51984_) [n:127.0.0.1:51984_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@b8862b9 name:ZooKeeperConnection Watcher:127.0.0.1:33903/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727954 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 727959 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 728966 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51984_
   [junit4]   2> 728970 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:51984_
   [junit4]   2> 728973 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 728974 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94384023632936969-127.0.0.1:51984_-n_0000000001
   [junit4]   2> 728975 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94384023632936964-127.0.0.1:40503_-n_0000000000 to know if I could be the leader
   [junit4]   2> 728994 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 728995 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 728995 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/cores
   [junit4]   2> 728996 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.SolrCore Created CoreDescriptor: {config=solrconfig.xml, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/cores/collection1/, collection=collection1, coreNodeName=, shard=, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/cores/collection1, transient=false, dataDir=data/, name=collection1, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json}
   [junit4]   2> 728997 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_ABBC68D25DC3779C-001/shard-1-001/cores/collection1/
   [junit4]   2> 728997 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 728998 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1
   [junit4]   2> 728998 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[ABBC68D25DC3779C]) [n:127.0.0.1:51984_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 728999 INFO  (coreLoadExecutor-1114-thread-1-processing-n:127.0.0.1:51984_) [n:127.0.0.1:51984_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 728999 INFO  (coreLoadExecutor-1114-thread-1-processing-n:127.0.0.1:51984_) [n:127.0.0.1:51984_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 729000 INFO  (zkCallback-316-thread-1-processing-n:127.0.0.1:40503_) [n:127.0.0.1:40503_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 729000 INFO  (coreLoadExecutor-1114-thread-1-processing-n:127.0.0.1:51984_) [n:127.0.0.1:51984_ c:collection1   x:collection1] o.a.s.c.ZkController Registering watch for external collection collection1
   [junit4]   2> 729000 INFO  (coreLoadExecutor-1114-thread-1-processing-n:127.0.0.1:51984_) [n:127.0.0.1:51984_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch collection1
   [junit4]   2> 729000 INFO  (OverseerStateUpdate-94384023632936964-127.0.0.1:40503_-n_0000000000) [n:127.0.0.1:40503_    ] o.a.

[...truncated too long message...]

it4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 53 more
   [junit4]   2> 3910993 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[ABBC68D25DC3779C]-worker) [n:127.0.0.1:58433_ c:addReplicaColl s:shard2 r:core_node1 x:addReplicaColl_shard2_replica2] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Aug 21, 2015 4:31:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {a_i1=PostingsFormat(name=MockRandom), _version_=PostingsFormat(name=MockRandom), text=PostingsFormat(name=Direct), range_facet_l_dv=BlockTreeOrds(blocksize=128), intDefault=PostingsFormat(name=MockRandom), id=BlockTreeOrds(blocksize=128), range_facet_l=PostingsFormat(name=MockRandom), other_tl1=PostingsFormat(name=MockRandom), timestamp=PostingsFormat(name=MockRandom), range_facet_i_dv=PostingsFormat(name=MockRandom), multiDefault=FSTOrd50, a_t=PostingsFormat(name=MockRandom)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene50), timestamp=DocValuesFormat(name=Memory), range_facet_i_dv=DocValuesFormat(name=Memory)}, sim=DefaultSimilarity, locale=sv, timezone=GB
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=2,free=50065472,total=530579456
   [junit4]   2> NOTE: All tests run in this JVM: [TestUtils, TestSchemaNameResource, TestStressRecovery, TestConfigReload, TestRequestStatusCollectionAPI, QueryParsingTest, OutputWriterTest, OpenExchangeRatesOrgProviderTest, BigEndianAscendingWordSerializerTest, DirectUpdateHandlerOptimizeTest, TestDefaultStatsCache, DistributedIntervalFacetingTest, TestDistribDocBasedVersion, LeaderElectionTest, TestScoreJoinQPScore, SampleTest, SolrCoreCheckLockOnStartupTest, TestXIncludeConfig, XmlUpdateRequestHandlerTest, DistributedQueryElevationComponentTest, HLLSerializationTest, TestFileDictionaryLookup, TestFieldTypeCollectionResource, CoreAdminCreateDiscoverTest, HttpPartitionTest, TestConfigSets, SpatialHeatmapFacetsTest, CollectionTooManyReplicasTest, TestMaxScoreQueryParser, HdfsBasicDistributedZk2Test, RemoteQueryErrorTest, SpellCheckCollatorTest, PathHierarchyTokenizerFactoryTest, TestSolrQueryParser, TestDefaultSimilarityFactory, TestPartialUpdateDeduplication, SimpleFacetsTest, TestCollationFieldDocValues, TestLeaderElectionZkExpiry, SortByFunctionTest, SparseHLLTest, TestZkChroot, TestFastOutputStream, SuggesterTest, TestBulkSchemaAPI, BlockDirectoryTest, SearchHandlerTest, CustomCollectionTest, DistributedFacetPivotLargeTest, TestNoOpRegenerator, TestClusterStateMutator, TestCSVResponseWriter, TestCloudPivotFacet, TestBinaryResponseWriter, TestSimpleTrackingShardHandler, SSLMigrationTest, TestSolrXml, RegexBoostProcessorTest, HighlighterMaxOffsetTest, DistributedDebugComponentTest, MoreLikeThisHandlerTest, TestScoreJoinQPNoScore, HardAutoCommitTest, FullSolrCloudDistribCmdsTest, SolrIndexSplitterTest, SolrInfoMBeanTest, BadIndexSchemaTest, TestExtendedDismaxParser, TestRemoveLastDynamicCopyField, MergeStrategyTest, TestInfoStreamLogging, TestRestoreCore, AutoCommitTest, TestSchemaVersionResource, DocExpirationUpdateProcessorFactoryTest, TestFreeTextSuggestions, TestTrackingShardHandlerFactory, VersionInfoTest, ChangedSchemaMergeTest, ExplicitHLLTest, TestSolrJ, TestMacros, TestLuceneMatchVersion, BasicDistributedZk2Test, UnloadDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, TestRandomFaceting, ZkCLITest, ShardRoutingCustomTest, TestFaceting, TestRecovery, TestHashPartitioner, TermVectorComponentDistributedTest, ZkControllerTest, TestRealTimeGet, TestStressReorder, TestJoin, TestReload, SolrCoreTest, StatsComponentTest, TestGroupingSearch, PeerSyncTest, ConvertedLegacyTest, TestSort, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores, DirectUpdateHandlerTest, TestBadConfig, OverseerCollectionProcessorTest, TestIndexSearcher, HighlighterTest, SoftAutoCommitTest, AnalysisAfterCoreReloadTest, CoreAdminHandlerTest, DocValuesTest, TestTrie, SpatialFilterTest, SuggesterWFSTTest, PolyFieldTest, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, TestRemoteStreaming, DocValuesMultiTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, LukeRequestHandlerTest, TestQueryUtils, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestQueryTypes, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, RequestHandlersTest, DocumentBuilderTest, TermVectorComponentTest, FastVectorHighlighterTest, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, SolrPluginUtilsTest, TestCollationField, MBeansHandlerTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, TestLFUCache, SOLR749Test, UpdateParamsTest, TestSolrIndexConfig, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, SolrIndexConfigTest, TestSolrDeletionPolicy2, MultiTermTest, TestDocSet, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestCodecSupport, SynonymTokenizerTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestBM25SimilarityFactory, TestFastWriter, ResourceLoaderTest, ScriptEngineTest, URLClassifyProcessorTest, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, AssignTest, AsyncCallRequestStatusResponseTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, DeleteInactiveReplicaTest, DistribCursorPagingTest, LeaderInitiatedRecoveryOnCommitTest, MultiThreadedOCPTest, OverseerStatusTest, ReplicationFactorTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverTest, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [479/531] on J2 in 634.63s, 1 test, 1 error <<< FAILURES!

[...truncated 168 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:733: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build.xml:230: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/common-build.xml:524: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1449: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1003: There were test failures: 531 suites, 2104 tests, 2 errors, 1 failure, 27 ignored (15 assumptions)

Total time: 202 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.3 - Build # 11 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.3/11/

4 tests failed.
REGRESSION:  org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test.test

Error Message:
Server refused connection at: http://127.0.0.1:42930/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:42930/collection1
	at __randomizedtesting.SeedInfo.seed([D18E239D34220062:59DA1C479ADE6D9A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:567)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:365)
	at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:98)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:465)
	... 48 more


REGRESSION:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1440111752853,generation=2,filelist=[_785.fdt, _785.fdx, _785.fnm, _785.nvd, _785.nvm, _785.si, _785_Memory_0.ram, _ec5.cfe, _ec5.cfs, _ec5.si, _g2j.cfe, _g2j.cfs, _g2j.si, _g2k.cfe, _g2k.cfs, _g2k.si, _g2l.cfe, _g2l.cfs, _g2l.si, _g2m.cfe, _g2m.cfs, _g2m.si, _g2n.cfe, _g2n.cfs, _g2n.si, segments_2]}]> but was:<[{indexVersion=1440111752853,generation=2,filelist=[_785.fdt, _785.fdx, _785.fnm, _785.nvd, _785.nvm, _785.si, _785_Memory_0.ram, _ec5.cfe, _ec5.cfs, _ec5.si, _g2j.cfe, _g2j.cfs, _g2j.si, _g2k.cfe, _g2k.cfs, _g2k.si, _g2l.cfe, _g2l.cfs, _g2l.si, _g2m.cfe, _g2m.cfs, _g2m.si, _g2n.cfe, _g2n.cfs, _g2n.si, segments_2]}, {indexVersion=1440111752853,generation=3,filelist=[_785.fdt, _785.fdx, _785.fnm, _785.nvd, _785.nvm, _785.si, _785_Memory_0.ram, _ec5.cfe, _ec5.cfs, _ec5.si, _g2o.cfe, _g2o.cfs, _g2o.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1440111752853,generation=2,filelist=[_785.fdt, _785.fdx, _785.fnm, _785.nvd, _785.nvm, _785.si, _785_Memory_0.ram, _ec5.cfe, _ec5.cfs, _ec5.si, _g2j.cfe, _g2j.cfs, _g2j.si, _g2k.cfe, _g2k.cfs, _g2k.si, _g2l.cfe, _g2l.cfs, _g2l.si, _g2m.cfe, _g2m.cfs, _g2m.si, _g2n.cfe, _g2n.cfs, _g2n.si, segments_2]}]> but was:<[{indexVersion=1440111752853,generation=2,filelist=[_785.fdt, _785.fdx, _785.fnm, _785.nvd, _785.nvm, _785.si, _785_Memory_0.ram, _ec5.cfe, _ec5.cfs, _ec5.si, _g2j.cfe, _g2j.cfs, _g2j.si, _g2k.cfe, _g2k.cfs, _g2k.si, _g2l.cfe, _g2l.cfs, _g2l.si, _g2m.cfe, _g2m.cfs, _g2m.si, _g2n.cfe, _g2n.cfs, _g2n.si, segments_2]}, {indexVersion=1440111752853,generation=3,filelist=[_785.fdt, _785.fdx, _785.fnm, _785.nvd, _785.nvm, _785.si, _785_Memory_0.ram, _ec5.cfe, _ec5.cfs, _ec5.si, _g2o.cfe, _g2o.cfs, _g2o.si, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([D18E239D34220062:F45938AD446A0E61]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1138)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=36400, name=collection3, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=36400, name=collection3, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:56591: Could not find collection : awholynewstresscollection_collection3_2
	at __randomizedtesting.SeedInfo.seed([D18E239D34220062]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=45779, name=collection0, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=45779, name=collection0, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45317/c_ctq: Could not find collection : awholynewstresscollection_collection0_5
	at __randomizedtesting.SeedInfo.seed([D18E239D34220062]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)




Build Log:
[...truncated 10548 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/init-core-data-001
   [junit4]   2> 322563 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 323185 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 323891 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 324074 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 324084 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 324169 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 324223 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_50952_hdfs____.jo7zuj/webapp
   [junit4]   2> 324423 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 325558 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:50952
   [junit4]   2> 326381 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 326389 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 326402 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46876_datanode____39jbng/webapp
   [junit4]   2> 326555 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 327280 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46876
   [junit4]   2> 327381 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 327384 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 327398 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_41083_datanode____.5pug7f/webapp
   [junit4]   2> 327528 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 328078 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[D18E239D34220062]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41083
   [junit4]   2> 328174 INFO  (IPC Server handler 4 on 36136) [    ] BlockStateChange BLOCK* processReport: from storage DS-f0872738-e0c0-4d01-9478-0c1dc3ebcb88 node DatanodeRegistration(127.0.0.1, datanodeUuid=9b9657b3-b7e6-4897-ad61-cc227bd20014, infoPort=46876, ipcPort=53362, storageInfo=lv=-56;cid=testClusterID;nsid=1153788544;c=0), blocks: 0, hasStaleStorages: true, processing time: 3 msecs
   [junit4]   2> 328176 INFO  (IPC Server handler 4 on 36136) [    ] BlockStateChange BLOCK* processReport: from storage DS-8cd73937-4371-4a6c-8d07-eef52425bcc2 node DatanodeRegistration(127.0.0.1, datanodeUuid=9b9657b3-b7e6-4897-ad61-cc227bd20014, infoPort=46876, ipcPort=53362, storageInfo=lv=-56;cid=testClusterID;nsid=1153788544;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 328310 INFO  (IPC Server handler 2 on 36136) [    ] BlockStateChange BLOCK* processReport: from storage DS-b8701070-79d4-49b0-aa1f-ba52104cf11b node DatanodeRegistration(127.0.0.1, datanodeUuid=5bd720f6-778f-4a21-b6ed-77c036cff068, infoPort=41083, ipcPort=58073, storageInfo=lv=-56;cid=testClusterID;nsid=1153788544;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 328311 INFO  (IPC Server handler 2 on 36136) [    ] BlockStateChange BLOCK* processReport: from storage DS-9a879207-659e-4df8-adc8-73f601baedd9 node DatanodeRegistration(127.0.0.1, datanodeUuid=5bd720f6-778f-4a21-b6ed-77c036cff068, infoPort=41083, ipcPort=58073, storageInfo=lv=-56;cid=testClusterID;nsid=1153788544;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 329054 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 329054 INFO  (Thread-4671) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 329055 INFO  (Thread-4671) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 329155 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.ZkTestServer start zk server on port:47501
   [junit4]   2> 329155 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 329156 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 329159 INFO  (zkCallback-147-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3c26f59c name:ZooKeeperConnection Watcher:127.0.0.1:47501 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 329159 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 329160 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 329160 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 329163 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 329164 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 329166 INFO  (zkCallback-148-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@11dbdc59 name:ZooKeeperConnection Watcher:127.0.0.1:47501/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 329166 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 329167 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 329167 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 329171 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 329172 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 329174 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 329176 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 329177 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 329179 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 329180 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 329182 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 329182 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 329184 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 329185 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 329189 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 329189 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 329192 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 329192 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 329195 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 329195 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 329197 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 329198 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 329203 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 329203 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 329205 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 329206 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 329208 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 329208 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 329513 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/cores/collection1
   [junit4]   2> 329515 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 329517 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@65bb52be{/,null,AVAILABLE}
   [junit4]   2> 329518 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4e4d9f3c{HTTP/1.1}{127.0.0.1:47540}
   [junit4]   2> 329518 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.e.j.s.Server Started @332465ms
   [junit4]   2> 329518 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=47540, hostContext=/, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/cores, solr.data.dir=hdfs://localhost:36136/hdfs__localhost_36136__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.3_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001_tempDir-002_control_data}
   [junit4]   2> 329519 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@1fe91485
   [junit4]   2> 329519 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/'
   [junit4]   2> 329537 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 329537 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 329540 INFO  (zkCallback-149-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2514cd52 name:ZooKeeperConnection Watcher:127.0.0.1:47501/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 329540 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 329541 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 329543 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 329544 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/solr.xml
   [junit4]   2> 329561 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/cores
   [junit4]   2> 329561 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.CoreContainer New CoreContainer 1877121038
   [junit4]   2> 329562 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/]
   [junit4]   2> 329562 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/lib
   [junit4]   2> 329562 WARN  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/lib).
   [junit4]   2> 329573 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 329574 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 329575 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 329575 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 329575 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47501/solr
   [junit4]   2> 329576 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 329576 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 329576 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 329578 INFO  (zkCallback-151-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6fabaf02 name:ZooKeeperConnection Watcher:127.0.0.1:47501 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 329579 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 329579 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 329581 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 329583 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@48aa33e6 name:ZooKeeperConnection Watcher:127.0.0.1:47501/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 329583 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 329584 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 329587 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 329589 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 329591 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 329594 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 329596 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 329598 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 329600 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 329601 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 329603 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47540_
   [junit4]   2> 329603 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:47540_
   [junit4]   2> 329605 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 329607 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 329608 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 329609 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94378782414798852-127.0.0.1:47540_-n_0000000000
   [junit4]   2> 329610 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:47540_
   [junit4]   2> 329610 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 329612 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.Overseer Overseer (id=94378782414798852-127.0.0.1:47540_-n_0000000000) starting
   [junit4]   2> 329614 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 329621 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 329621 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 329621 INFO  (OverseerCollectionProcessor-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.OverseerCollectionProcessor Process current queue of collection creations
   [junit4]   2> 329622 INFO  (OverseerStateUpdate-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 329638 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 329638 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 329639 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/cores
   [junit4]   2> 329640 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.SolrCore Created CoreDescriptor: {coreNodeName=, configSetProperties=configsetprops.json, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/cores/collection1, collection=control_collection, schema=schema.xml, dataDir=data/, transient=false, config=solrconfig.xml, loadOnStartup=true, name=collection1, shard=, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/cores/collection1/}
   [junit4]   2> 329641 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/cores/collection1/
   [junit4]   2> 329641 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 329642 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 329642 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 329643 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 329643 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2
   [junit4]   2> 329643 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 329645 INFO  (OverseerStateUpdate-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:47540",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "node_name":"127.0.0.1:47540_",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "collection":"control_collection"} current state version: 0
   [junit4]   2> 329646 INFO  (OverseerStateUpdate-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:47540",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "node_name":"127.0.0.1:47540_",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "collection":"control_collection"}
   [junit4]   2> 329646 INFO  (OverseerStateUpdate-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 329646 INFO  (OverseerStateUpdate-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 329648 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 329643 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [n:127.0.0.1:47540_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 329650 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 1
   [junit4]   2> 330643 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 330643 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 330644 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 330644 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 330645 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 330645 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/cores/collection1/'
   [junit4]   2> 330662 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 330671 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 330706 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.3.0
   [junit4]   2> 330739 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 330740 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 330762 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 331037 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 331047 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 331049 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 331063 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 331068 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 331072 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 331073 WARN  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 331073 WARN  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 331073 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 331074 WARN  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 331075 WARN  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 331076 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2
   [junit4]   2> 331076 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 331076 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 331086 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:36136/solr_hdfs_home
   [junit4]   2> 331086 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 331087 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 331087 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e47f5ae
   [junit4]   2> 331087 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36136/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 331154 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:36136/solr_hdfs_home/control_collection/core_node1/data for locality metrics.
   [junit4]   2> 331155 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:36136/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 331160 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:36136/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 331197 WARN  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:36136/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 331197 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:36136/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 331219 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 331219 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 331219 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 331259 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 331259 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:36136/solr_hdfs_home/control_collection/core_node1/data/index for locality metrics.
   [junit4]   2> 331259 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:36136/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 331260 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=9, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=45.9296875, floorSegmentMB=1.669921875, forceMergeDeletesPctAllowed=8.6685309695476, segmentsPerTier=14.0, maxCFSSegmentSizeMB=1.5888671875, noCFSRatio=0.4069667879198541
   [junit4]   2> 331596 INFO  (IPC Server handler 3 on 36136) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41457 is added to blk_1073741825_1001{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-9a879207-659e-4df8-adc8-73f601baedd9:NORMAL:127.0.0.1:41457|RBW], ReplicaUnderConstruction[[DISK]DS-f0872738-e0c0-4d01-9478-0c1dc3ebcb88:NORMAL:127.0.0.1:40587|RBW]]} size 71
   [junit4]   2> 331598 INFO  (IPC Server handler 2 on 36136) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40587 is added to blk_1073741825_1001 size 71
   [junit4]   2> 332005 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b65bf9fe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de200ff); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 332005 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 332009 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:36136/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 332015 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 332016 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 332016 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 332016 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 332017 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 332017 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 332017 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 332018 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 332018 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 332021 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 332023 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 332024 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 332026 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 332033 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/plugins,/admin/mbeans,/admin/properties,/admin/luke,/admin/segments,/update/csv,/update,/admin/threads,/update/json/docs,/admin/file,/replication,/schema,/update/json,/admin/ping,/admin/system,/get,standard,/config,/admin/logging
   [junit4]   2> 332036 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 332040 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 332040 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing HdfsUpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 tlogDfsReplication=2
   [junit4]   2> 332060 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 332060 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 332064 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@64c5bd74
   [junit4]   2> 332149 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b65bf9fe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de200ff); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 332149 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 332150 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@71c59d1b[collection1] main
   [junit4]   2> 332150 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 332151 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 332151 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 332151 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 332152 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 332152 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 332152 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 332153 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 332153 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 332153 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 332156 INFO  (searcherExecutor-610-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@71c59d1b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 332162 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 332162 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version buckets from index
   [junit4]   2> 332163 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 332163 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1510060521894707200
   [junit4]   2> 332163 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1 ms to seed version buckets with highest version 1510060521894707200
   [junit4]   2> 332163 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 332164 INFO  (coreLoadExecutor-609-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 332165 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:47540 collection:control_collection shard:shard1
   [junit4]   2> 332166 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 332170 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94378782414798852-core_node1-n_0000000000
   [junit4]   2> 332172 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 332174 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 332175 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 332175 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 332175 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:47540/collection1/
   [junit4]   2> 332175 INFO  (OverseerStateUpdate-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 332175 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 332175 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:47540/collection1/ has no replicas
   [junit4]   2> 332176 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:47540/collection1/ shard1
   [junit4]   2> 332176 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 332177 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 332179 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 2
   [junit4]   2> 332189 INFO  (OverseerStateUpdate-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:47540",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 332192 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 332193 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 332195 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 332232 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:47540/collection1/ and leader is http://127.0.0.1:47540/collection1/
   [junit4]   2> 332232 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47540
   [junit4]   2> 332232 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 332232 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 332233 INFO  (coreZkRegister-603-thread-1-processing-r:core_node1 n:127.0.0.1:47540_ x:collection1 s:shard1 c:control_collection) [n:127.0.0.1:47540_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 332234 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 332236 INFO  (OverseerStateUpdate-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "base_url":"http://127.0.0.1:47540",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "node_name":"127.0.0.1:47540_",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "collection":"control_collection"} current state version: 3
   [junit4]   2> 332236 INFO  (OverseerStateUpdate-94378782414798852-127.0.0.1:47540_-n_0000000000) [n:127.0.0.1:47540_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "base_url":"http://127.0.0.1:47540",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "node_name":"127.0.0.1:47540_",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "collection":"control_collection"}
   [junit4]   2> 332348 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 332350 INFO  (zkCallback-152-thread-1-processing-n:127.0.0.1:47540_) [n:127.0.0.1:47540_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 332651 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 332652 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 332653 INFO  (zkCallback-154-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@47f46dad name:ZooKeeperConnection Watcher:127.0.0.1:47501/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 332653 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 332654 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 332654 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 332658 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 332979 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/shard-1-001/cores/collection1
   [junit4]   2> 332980 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/shard-1-001
   [junit4]   2> 332981 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 332984 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3380db19{/,null,AVAILABLE}
   [junit4]   2> 332984 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6b9e3b11{HTTP/1.1}{127.0.0.1:34628}
   [junit4]   2> 332985 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.e.j.s.Server Started @335932ms
   [junit4]   2> 332985 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:36136/hdfs__localhost_36136__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.3_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/shard-1-001/cores, hostPort=34628}
   [junit4]   2> 332985 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@1fe91485
   [junit4]   2> 332985 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/shard-1-001/'
   [junit4]   2> 333003 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 333005 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 333006 INFO  (zkCallback-155-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1e102798 name:ZooKeeperConnection Watcher:127.0.0.1:47501/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 333006 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 333006 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 333008 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 333009 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/shard-1-001/solr.xml
   [junit4]   2> 333025 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/shard-1-001/cores
   [junit4]   2> 333025 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.CoreContainer New CoreContainer 1900809906
   [junit4]   2> 333025 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/shard-1-001/]
   [junit4]   2> 333026 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/shard-1-001/lib
   [junit4]   2> 333026 WARN  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_D18E239D34220062-001/shard-1-001/lib).
   [junit4]   2> 333036 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 333037 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 333038 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 333038 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 333038 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47501/solr
   [junit4]   2> 333038 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 333038 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 333039 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 333041 INFO  (zkCallback-157-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7c2b750b name:ZooKeeperConnection Watcher:127.0.0.1:47501 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 333041 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 333042 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[D18E239D34220062]) [    ] o.a.s.c.c.SolrZkClient Using default ZkAC

[...truncated too long message...]

(TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001/solr-instance-027/./collection1/data/index [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001/solr-instance-027/./collection1/data/index;done=false>>]
   [junit4]   2> 6447424 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001/solr-instance-027/./collection1/data/index
   [junit4]   2> 6447424 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@14210e86{/solr,null,UNAVAILABLE}
   [junit4]   2> 6447426 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@7fa6dbaa{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 6447426 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1856313456
   [junit4]   2> 6447426 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@12ed078b
   [junit4]   2> 6447427 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.u.UpdateHandler closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 6447427 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.u.SolrCoreState Closing SolrCoreState
   [junit4]   2> 6447427 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 6447427 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.u.DefaultSolrCoreState closing IndexWriter with IndexWriterCloser
   [junit4]   2> 6447428 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on request.
   [junit4]   2> 6447428 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 6447428 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001/solr-instance-028/./collection1/data [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001/solr-instance-028/./collection1/data;done=false>>]
   [junit4]   2> 6447429 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001/solr-instance-028/./collection1/data
   [junit4]   2> 6447429 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001/solr-instance-028/./collection1/data/index [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001/solr-instance-028/./collection1/data/index;done=false>>]
   [junit4]   2> 6447429 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001/solr-instance-028/./collection1/data/index
   [junit4]   2> 6447430 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[D18E239D34220062]) [    x:collection1] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5910c85d{/solr,null,UNAVAILABLE}
   [junit4]   2> 6447431 INFO  (SUITE-TestReplicationHandler-seed#[D18E239D34220062]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_D18E239D34220062-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {newname=PostingsFormat(name=Memory doPackFST= false), name=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=Memory doPackFST= false)}, docValues:{}, sim=DefaultSimilarity, locale=ar_OM, timezone=IST
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=1,free=57398664,total=531103744
   [junit4]   2> NOTE: All tests run in this JVM: [TestXIncludeConfig, SimpleCollectionCreateDeleteTest, OverseerStatusTest, DataDrivenBlockJoinTest, TestReplicaProperties, TestConfigReload, BinaryUpdateRequestHandlerTest, TestJsonRequest, XsltUpdateRequestHandlerTest, TestComplexPhraseQParserPlugin, DistributedDebugComponentTest, DistribCursorPagingTest, SparseHLLTest, MergeStrategyTest, VersionInfoTest, TestFastOutputStream, SolrInfoMBeanTest, TestUtils, TestDocSet, TestDefaultSimilarityFactory, DistributedIntervalFacetingTest, HdfsBasicDistributedZk2Test, SortByFunctionTest, SpatialHeatmapFacetsTest, BadIndexSchemaTest, TestSolrCLIRunExample, DistributedFacetPivotLargeTest, HighlighterMaxOffsetTest, LeaderElectionTest, FileBasedSpellCheckerTest, TestCollationField, StatsComponentTest, SOLR749Test, TestSystemIdResolver, RulesTest, BlockCacheTest, TestBM25SimilarityFactory, TestSolrQueryParser, DocExpirationUpdateProcessorFactoryTest, LoggingHandlerTest, TestLeaderElectionZkExpiry, ExplicitHLLTest, SuggesterTest, MoreLikeThisHandlerTest, TestSurroundQueryParser, TestDistribDocBasedVersion, ResponseHeaderTest, QueryParsingTest, TestZkChroot, TestNoOpRegenerator, TestSort, TestConfigSets, OverseerTest, TestExactSharedStatsCache, SpellCheckCollatorTest, RecoveryZkTest, TermVectorComponentDistributedTest, TestMacros, TestReload, SharedFSAutoReplicaFailoverTest, DistributedQueryComponentCustomSortTest, TestLuceneMatchVersion, CachingDirectoryFactoryTest, DistributedExpandComponentTest, TestExpandComponent, TestExtendedDismaxParser, TestCSVResponseWriter, TestSimpleTrackingShardHandler, TestPartialUpdateDeduplication, XmlUpdateRequestHandlerTest, TestManagedStopFilterFactory, CoreAdminCreateDiscoverTest, SearchHandlerTest, TestCloudPivotFacet, TestFieldTypeCollectionResource, TestScoreJoinQPNoScore, TestStressReorder, TestRandomRequestDistribution, SolrCoreCheckLockOnStartupTest, DOMUtilTest, TestTrackingShardHandlerFactory, TestBinaryResponseWriter, TestSolrXml, TestGroupingSearch, TestInfoStreamLogging, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler]
   [junit4] Completed [531/531] on J2 in 4238.84s, 13 tests, 1 failure <<< FAILURES!

[...truncated 14 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:733: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build.xml:230: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/common-build.xml:524: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1449: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1003: There were test failures: 531 suites, 2104 tests, 3 errors, 1 failure, 27 ignored (15 assumptions)

Total time: 376 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.3 - Build # 10 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.3/10/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space


REGRESSION:  org.apache.solr.cloud.RollingRestartTest.test

Error Message:
Unable to restart (#0): CloudJettyRunner [url=http://127.0.0.1:41463/collection1]

Stack Trace:
java.lang.AssertionError: Unable to restart (#0): CloudJettyRunner [url=http://127.0.0.1:41463/collection1]
	at __randomizedtesting.SeedInfo.seed([19C2D7FD50A4B9B2:9196E827FE58D44A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:104)
	at org.apache.solr.cloud.RollingRestartTest.test(RollingRestartTest.java:53)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at java.lang.Class.getDeclaredFields0(Native Method)
	at java.lang.Class.privateGetDeclaredFields(Class.java:2499)
	at java.lang.Class.privateGetPublicFields(Class.java:2530)
	at java.lang.Class.getFields(Class.java:1420)
	at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.determineType(JmxMonitoredMap.java:323)
	at org.apache.solr.core.JmxMonitoredMap$SolrDynamicMBean.getMBeanInfo(JmxMonitoredMap.java:298)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:644)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at javax.management.AttributeValueExp.getAttribute(AttributeValueExp.java:163)
	at javax.management.AttributeValueExp.apply(AttributeValueExp.java:105)
	at javax.management.BinaryRelQueryExp.apply(BinaryRelQueryExp.java:109)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.filterListOfObjectInstances(DefaultMBeanServerInterceptor.java:1626)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:545)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:619)
	at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:144)
	at org.apache.solr.core.SolrCore.close(SolrCore.java:1246)
	at org.apache.solr.core.SolrCores.close(SolrCores.java:124)
	at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:562)
	at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:172)
	at org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:198)
	at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:552)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.destroyServers(AbstractFullDistribZkTestBase.java:1509)
	at org.apache.solr.BaseDistributedSearchTestCase.distribTearDown(BaseDistributedSearchTestCase.java:311)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.distribTearDown(AbstractDistribZkTestBase.java:255)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribTearDown(AbstractFullDistribZkTestBase.java:1487)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=185908, name=collection1, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=185908, name=collection1, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33639/p_g/e: Could not find collection : awholynewstresscollection_collection1_0
	at __randomizedtesting.SeedInfo.seed([19C2D7FD50A4B9B2]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)




Build Log:
[...truncated 10755 lines...]
   [junit4] Suite: org.apache.solr.cloud.RollingRestartTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/init-core-data-001
   [junit4]   2> 1297697 INFO  (SUITE-RollingRestartTest-seed#[19C2D7FD50A4B9B2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1297697 INFO  (SUITE-RollingRestartTest-seed#[19C2D7FD50A4B9B2]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1297703 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1297705 INFO  (Thread-2479) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1297705 INFO  (Thread-2479) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1297805 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.ZkTestServer start zk server on port:33002
   [junit4]   2> 1297805 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1297825 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1297833 INFO  (zkCallback-750-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@301210dd name:ZooKeeperConnection Watcher:127.0.0.1:33002 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1297833 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1297834 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1297834 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1297838 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1297846 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1297849 INFO  (zkCallback-751-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@20ebee3d name:ZooKeeperConnection Watcher:127.0.0.1:33002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1297850 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1297850 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1297850 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1297853 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1297862 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1297865 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1297870 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1297870 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1297873 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1297874 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1297875 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1297876 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1297877 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1297877 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1297879 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1297880 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1297881 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1297881 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1297883 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1297884 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1297885 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1297886 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1297887 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1297887 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1297890 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1297890 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1297891 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1297892 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1298414 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1
   [junit4]   2> 1298429 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 1298437 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1ca1635e{/,null,AVAILABLE}
   [junit4]   2> 1298442 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.e.j.s.ServerConnector Started ServerConnector@78c9eeea{HTTP/1.1}{127.0.0.1:55078}
   [junit4]   2> 1298444 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.e.j.s.Server Started @1301636ms
   [junit4]   2> 1298444 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores, hostPort=55078, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/tempDir-001/control/data}
   [junit4]   2> 1298445 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0add57
   [junit4]   2> 1298445 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/'
   [junit4]   2> 1298470 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1298471 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1298492 INFO  (zkCallback-752-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3016df66 name:ZooKeeperConnection Watcher:127.0.0.1:33002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1298494 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1298494 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1298496 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1298496 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/solr.xml
   [junit4]   2> 1298514 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores
   [junit4]   2> 1298515 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.CoreContainer New CoreContainer 1851165921
   [junit4]   2> 1298515 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/]
   [junit4]   2> 1298515 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/lib
   [junit4]   2> 1298517 WARN  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/lib).
   [junit4]   2> 1298527 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1298529 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1298530 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1298530 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1298530 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33002/solr
   [junit4]   2> 1298530 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1298531 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1298542 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1298548 INFO  (zkCallback-754-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2677c5a2 name:ZooKeeperConnection Watcher:127.0.0.1:33002 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1298550 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1298550 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1298564 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1298578 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@557a8454 name:ZooKeeperConnection Watcher:127.0.0.1:33002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1298578 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1298580 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1298583 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1298585 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1298589 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1298591 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1298594 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1298596 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1298599 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1298601 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1298602 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55078_
   [junit4]   2> 1298603 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:55078_
   [junit4]   2> 1298604 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1298605 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1298608 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1298608 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94372999510491140-127.0.0.1:55078_-n_0000000000
   [junit4]   2> 1298610 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55078_
   [junit4]   2> 1298611 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1298614 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.Overseer Overseer (id=94372999510491140-127.0.0.1:55078_-n_0000000000) starting
   [junit4]   2> 1298617 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1298626 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1298629 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1298648 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1298649 INFO  (OverseerCollectionProcessor-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.OverseerCollectionProcessor Process current queue of collection creations
   [junit4]   2> 1298666 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1298666 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1298667 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores
   [junit4]   2> 1298669 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.SolrCore Created CoreDescriptor: {instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1, transient=false, configSetProperties=configsetprops.json, loadOnStartup=true, coreNodeName=, shard=, dataDir=data/, config=solrconfig.xml, name=collection1, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/, collection=control_collection, schema=schema.xml}
   [junit4]   2> 1298669 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/
   [junit4]   2> 1298670 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 1298671 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1298671 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1298677 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1298677 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1298678 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1
   [junit4]   2> 1298678 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:55078_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1298680 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "node_name":"127.0.0.1:55078_",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1:55078",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "roles":null,
   [junit4]   2>   "core":"collection1"} current state version: 0
   [junit4]   2> 1298682 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "node_name":"127.0.0.1:55078_",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1:55078",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "roles":null,
   [junit4]   2>   "core":"collection1"}
   [junit4]   2> 1298682 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1298682 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1298685 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1298689 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 1
   [junit4]   2> 1299677 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1299678 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1299678 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1299678 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 1299679 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1299679 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/'
   [junit4]   2> 1299711 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1299720 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1299746 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.3.0
   [junit4]   2> 1299792 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1299794 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1299803 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1300333 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1300356 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1300360 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1300381 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1300390 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1300394 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1300405 WARN  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1300406 WARN  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1300406 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1300410 WARN  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1300410 WARN  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1300411 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1
   [junit4]   2> 1300411 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1300412 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore solr.StandardDirectoryFactory
   [junit4]   2> 1300412 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1300412 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@262d06f4
   [junit4]   2> 1300413 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/data
   [junit4]   2> 1300413 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1300414 WARN  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1300415 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/data/index
   [junit4]   2> 1300415 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=4, maxMergedSegmentMB=1.8974609375, floorSegmentMB=1.89453125, forceMergeDeletesPctAllowed=17.737388939352694, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.300668987798983
   [junit4]   2> 1300421 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1300421 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 1300429 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/data/
   [junit4]   2> 1300431 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1300432 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1300432 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1300432 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1300432 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1300433 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1300433 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1300433 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1300433 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1300437 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1300439 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1300440 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1300442 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1300446 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/threads,/admin/ping,/admin/system,/admin/luke,/admin/file,/update/json,standard,/admin/plugins,/update,/update/csv,/admin/segments,/replication,/admin/logging,/schema,/update/json/docs,/config,/admin/mbeans,/admin/properties,/get
   [junit4]   2> 1300448 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1300449 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1300449 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1300450 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 1300450 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 1300450 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=9, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18667651172351657]
   [junit4]   2> 1300451 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1300451 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 1300451 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@63e4780[collection1] main
   [junit4]   2> 1300452 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 1300452 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1300453 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1300453 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1300453 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1300453 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1300454 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1300455 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1300455 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1300455 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1300456 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1300457 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1300458 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1300458 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1509967995053539328
   [junit4]   2> 1300457 INFO  (searcherExecutor-2058-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@63e4780[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1300461 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 4 ms to seed version buckets with highest version 1509967995053539328
   [junit4]   2> 1300461 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1300464 INFO  (coreLoadExecutor-2057-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1300467 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:55078 collection:control_collection shard:shard1
   [junit4]   2> 1300471 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1300476 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94372999510491140-core_node1-n_0000000000
   [junit4]   2> 1300477 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 1300479 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1300480 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1300482 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1300483 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1300483 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55078/collection1/
   [junit4]   2> 1300483 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1300483 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:55078/collection1/ has no replicas
   [junit4]   2> 1300483 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55078/collection1/ shard1
   [junit4]   2> 1300483 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1300485 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1300487 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 2
   [junit4]   2> 1300489 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1300490 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:55078",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 1300595 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1300597 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 1300641 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:55078/collection1/ and leader is http://127.0.0.1:55078/collection1/
   [junit4]   2> 1300641 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55078
   [junit4]   2> 1300641 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1300641 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1300641 INFO  (coreZkRegister-2051-thread-1-processing-n:127.0.0.1:55078_ r:core_node1 c:control_collection x:collection1 s:shard1) [n:127.0.0.1:55078_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1300643 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1300644 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "node_name":"127.0.0.1:55078_",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1:55078",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "core":"collection1"} current state version: 3
   [junit4]   2> 1300645 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "node_name":"127.0.0.1:55078_",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1:55078",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "core":"collection1"}
   [junit4]   2> 1300680 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1300686 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1300687 INFO  (zkCallback-757-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@29d8fe6e name:ZooKeeperConnection Watcher:127.0.0.1:33002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1300687 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1300687 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1300687 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1300689 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1300689 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1300689 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1300696 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1300706 INFO  (zkCallback-758-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5818df49 name:ZooKeeperConnection Watcher:127.0.0.1:33002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1300707 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1300707 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1300712 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1300714 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 1300714 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1300715 INFO  (zkCallback-757-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1300715 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1300718 INFO  (zkCallback-757-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 1300722 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 1300832 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 1300833 INFO  (zkCallback-757-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1300833 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1300836 INFO  (zkCallback-757-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 5
   [junit4]   2> 1300838 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 5
   [junit4]   2> 1301244 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/cores/collection1
   [junit4]   2> 1301245 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001
   [junit4]   2> 1301246 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 1301249 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f8b2526{/,null,AVAILABLE}
   [junit4]   2> 1301251 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.e.j.s.ServerConnector Started ServerConnector@60a31aad{HTTP/1.1}{127.0.0.1:48952}
   [junit4]   2> 1301251 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.e.j.s.Server Started @1304443ms
   [junit4]   2> 1301251 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solrconfig=solrconfig.xml, hostPort=48952, hostContext=/, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/tempDir-001/jetty1, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/cores}
   [junit4]   2> 1301251 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0add57
   [junit4]   2> 1301252 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/'
   [junit4]   2> 1301283 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1301302 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1301304 INFO  (zkCallback-759-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@34483b51 name:ZooKeeperConnection Watcher:127.0.0.1:33002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1301304 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1301304 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1301307 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1301307 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/solr.xml
   [junit4]   2> 1301326 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/cores
   [junit4]   2> 1301327 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.CoreContainer New CoreContainer 38185753
   [junit4]   2> 1301327 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/]
   [junit4]   2> 1301328 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/lib
   [junit4]   2> 1301328 WARN  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/lib).
   [junit4]   2> 1301339 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1301341 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1301341 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1301342 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1301342 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33002/solr
   [junit4]   2> 1301342 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1301342 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1301358 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1301360 INFO  (zkCallback-761-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@69f4ed8c name:ZooKeeperConnection Watcher:127.0.0.1:33002 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1301361 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1301362 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1301364 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1301365 INFO  (zkCallback-762-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@39382d7f name:ZooKeeperConnection Watcher:127.0.0.1:33002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1301366 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1301372 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1302385 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48952_
   [junit4]   2> 1302385 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:48952_
   [junit4]   2> 1302390 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1302390 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94372999510491145-127.0.0.1:48952_-n_0000000001
   [junit4]   2> 1302391 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94372999510491140-127.0.0.1:55078_-n_0000000000 to know if I could be the leader
   [junit4]   2> 1302403 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1302404 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1302406 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/cores
   [junit4]   2> 1302407 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.SolrCore Created CoreDescriptor: {instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/cores/collection1, dataDir=data/, schema=schema.xml, shard=, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/cores/collection1/, config=solrconfig.xml, configSetProperties=configsetprops.json, coreNodeName=, loadOnStartup=true, name=collection1, collection=collection1, transient=false}
   [junit4]   2> 1302408 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1/temp/solr.cloud.RollingRestartTest_19C2D7FD50A4B9B2-001/shard-1-001/cores/collection1/
   [junit4]   2> 1302408 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 1302409 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J1
   [junit4]   2> 1302410 INFO  (TEST-RollingRestartTest.test-seed#[19C2D7FD50A4B9B2]) [n:127.0.0.1:48952_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1302414 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1302414 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1302415 INFO  (zkCallback-755-thread-1-processing-n:127.0.0.1:55078_) [n:127.0.0.1:55078_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1302416 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.ZkController Registering watch for external collection collection1
   [junit4]   2> 1302416 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch collection1
   [junit4]   2> 1302417 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "node_name":"127.0.0.1:48952_",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1:48952",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "roles":null,
   [junit4]   2>   "core":"collection1"} current state version: 5
   [junit4]   2> 1302417 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1302417 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "node_name":"127.0.0.1:48952_",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "base_url":"http://127.0.0.1:48952",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "roles":null,
   [junit4]   2>   "core":"collection1"}
   [junit4]   2> 1302418 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=2
   [junit4]   2> 1302418 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1302419 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Updating data for collection1 to ver 0 
   [junit4]   2> 1302419 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1302421 INFO  (OverseerStateUpdate-94372999510491140-127.0.0.1:55078_-n_0000000000) [n:127.0.0.1:55078_    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 1302422 INFO  (zkCallback-762-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1302423 INFO  (zkCallback-762-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_    ] o.a.s.c.c.ZkStateReader Updating data for collection1 to ver 1 
   [junit4]   2> 1303420 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1303420 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.ZkController Check for collection zkNode:collection1
   [junit4]   2> 1303421 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1303421 INFO  (coreLoadExecutor-2068-thread-1-processing-n:127.0.0.1:48952_) [n:127.0.0.1:48952_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/collection1
   [junit4]   2> 130342

[...truncated too long message...]

112)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:296)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:56)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 8700680 WARN  (SyncThread:0) [    ] o.a.z.s.p.FileTxnLog fsync-ing the write ahead log in SyncThread:0 took 2471ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 8700682 ERROR (zkCallback-1623-thread-11-processing-n:127.0.0.1:52157__nci%2Fy-EventThread) [n:127.0.0.1:52157__nci%2Fy c:corewithnocollection2 s:shard1 r:core_node1 x:corewithnocollection2] o.a.s.c.ZkController There was a problem finding the leader in zk:java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1036)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1009)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1600)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:466)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.access$100(ZkController.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:296)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:56)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 8701370 INFO  (zkCallback-1619-thread-23-processing-n:127.0.0.1:43264__nci%2Fy-EventThread) [n:127.0.0.1:43264__nci%2Fy    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3d658aa7 name:ZooKeeperConnection Watcher:127.0.0.1:35395/solr got event WatchedEvent state:Expired type:None path:null path:null type:None
   [junit4]   2> 8701370 INFO  (zkCallback-1632-thread-16-processing-n:127.0.0.1:52129__nci%2Fy-EventThread) [n:127.0.0.1:52129__nci%2Fy    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@868be1a name:ZooKeeperConnection Watcher:127.0.0.1:35395/solr got event WatchedEvent state:Expired type:None path:null path:null type:None
   [junit4]   2> 8701370 INFO  (zkCallback-1632-thread-16-processing-n:127.0.0.1:52129__nci%2Fy-EventThread) [n:127.0.0.1:52129__nci%2Fy    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 8702345 INFO  (zkCallback-1632-thread-16-processing-n:127.0.0.1:52129__nci%2Fy-EventThread) [n:127.0.0.1:52129__nci%2Fy    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {other_tl1=PostingsFormat(name=LuceneVarGapDocFreqInterval), _version_=PostingsFormat(name=LuceneVarGapDocFreqInterval), text=Lucene50(blocksize=128), a_t=PostingsFormat(name=Memory doPackFST= true), range_facet_l=PostingsFormat(name=Memory doPackFST= true), multiDefault=PostingsFormat(name=Memory doPackFST= true), range_facet_i_dv=PostingsFormat(name=Memory doPackFST= true), a_i1=PostingsFormat(name=Memory doPackFST= true), id=BlockTreeOrds(blocksize=128), timestamp=PostingsFormat(name=Memory doPackFST= true), range_facet_l_dv=BlockTreeOrds(blocksize=128), intDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{range_facet_i_dv=DocValuesFormat(name=Direct), timestamp=DocValuesFormat(name=Direct), range_facet_l_dv=DocValuesFormat(name=Lucene50)}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=sr_RS_#Latn, timezone=Iceland
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=291,free=43178256,total=467140608
   [junit4]   2> NOTE: All tests run in this JVM: [TestObjectReleaseTracker, SyncSliceTest, TermsComponentTest, IndexBasedSpellCheckerTest, ConnectionManagerTest, PrimitiveFieldTypeTest, TestStressLucene, PeerSyncTest, HighlighterTest, TestCryptoKeys, DocValuesMissingTest, TestFieldCollectionResource, TestReqParamsAPI, URLClassifyProcessorTest, TestBlendedInfixSuggestions, CollectionsAPIAsyncDistributedZkTest, EnumFieldTest, TestUpdate, TestBinaryField, HdfsDirectoryFactoryTest, LeaderInitiatedRecoveryOnCommitTest, SaslZkACLProviderTest, SolrCloudExampleTest, ParsingFieldUpdateProcessorsTest, BasicZkTest, TestRemoteStreaming, TestPivotHelperCode, TestSolrConfigHandler, TestRandomDVFaceting, TestIndexingPerformance, TestFaceting, TestMinMaxOnMultiValuedField, DistanceUnitsTest, DateFieldTest, HdfsUnloadDistributedZkTest, TestRangeQuery, SuggestComponentTest, TestDFRSimilarityFactory, TestRebalanceLeaders, TestArbitraryIndexDir, CoreAdminRequestStatusTest, ExitableDirectoryReaderTest, TestRawResponseWriter, TestExactStatsCache, HdfsThreadLeakTest, ExternalFileFieldSortTest, FullHLLTest, TestDocumentBuilder, TestDocBasedVersionConstraints, TestCloudInspectUtil, DateMathParserTest, FastVectorHighlighterTest, LeaderFailoverAfterPartitionTest, HLLUtilTest, TestSolrIndexConfig, TestManagedResource, AnalyticsMergeStrategyTest, DebugComponentTest, ConcurrentDeleteAndCreateCollectionTest, TestSolrDeletionPolicy1, TestChildDocTransformer, TestHighFrequencyDictionaryFactory, MultiTermTest, CleanupOldIndexTest, JavabinLoaderTest, DeleteReplicaTest, CSVRequestHandlerTest, CloudExitableDirectoryReaderTest, TestDynamicFieldCollectionResource, SolrIndexConfigTest, HdfsChaosMonkeySafeLeaderTest, DeleteLastCustomShardedReplicaTest, TestNamedUpdateProcessors, TestSearcherReuse, HighlighterConfigTest, TestHdfsUpdateLog, UniqFieldsUpdateProcessorFactoryTest, HdfsDirectoryTest, AddSchemaFieldsUpdateProcessorFactoryTest, HdfsRecoveryZkTest, ClusterStateTest, DeleteInactiveReplicaTest, ChaosMonkeyNothingIsSafeTest, TestReversedWildcardFilterFactory, TestCodecSupport, TestConfigOverlay, CloudMLTQParserTest, SegmentsInfoRequestHandlerTest, BasicDistributedZkTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.seed=19C2D7FD50A4B9B2 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sr_RS_#Latn -Dtests.timezone=Iceland -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4]   2> 8701370 INFO  (zkCallback-1619-thread-23-processing-n:127.0.0.1:43264__nci%2Fy-EventThread) [n:127.0.0.1:43264__nci%2Fy    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 8708129 INFO  (zkCallback-1619-thread-23-processing-n:127.0.0.1:43264__nci%2Fy-EventThread) [n:127.0.0.1:43264__nci%2Fy    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4] ERROR   0.00s J0 | CollectionsAPIDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4] Completed [531/531] on J0 in 6705.11s, 1 test, 2 errors <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/temp/junit4-J0-20150819_163311_681.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/heapdumps/java_pid27793.hprof ...
   [junit4] Heap dump file created [627916798 bytes in 10.806 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 13 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:733: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build.xml:230: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/common-build.xml:524: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1449: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1003: There were test failures: 531 suites, 2104 tests, 1 suite-level error, 2 errors, 1 failure, 26 ignored (14 assumptions)

Total time: 276 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.3 - Build # 9 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.3/9/

2 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Error from server at https://127.0.0.1:48730: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:48730: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:301)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:418)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:168)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=95103, name=collection4, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=95103, name=collection4, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:54815/t_ek/oe: Could not find collection : awholynewstresscollection_collection4_0
	at __randomizedtesting.SeedInfo.seed([6B115E9977A97773]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)




Build Log:
[...truncated 11499 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/init-core-data-001
   [junit4]   2> 2785491 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[6B115E9977A97773]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 2785491 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[6B115E9977A97773]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2785496 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2785497 INFO  (Thread-45712) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2785497 INFO  (Thread-45712) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2785597 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.ZkTestServer start zk server on port:40584
   [junit4]   2> 2785597 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2785598 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2785601 INFO  (zkCallback-1427-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@359fa838 name:ZooKeeperConnection Watcher:127.0.0.1:40584 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2785602 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2785602 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2785602 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 2785607 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2785607 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2785611 INFO  (zkCallback-1428-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c89e409 name:ZooKeeperConnection Watcher:127.0.0.1:40584/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2785611 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2785612 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2785612 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 2785614 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 2785616 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 2785618 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 2785619 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2785620 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2785622 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2785623 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 2785625 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2785625 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2785631 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2785631 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2785633 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2785633 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2785635 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2785636 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 2785638 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2785638 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2785640 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2785640 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2785642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2785643 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2785645 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2785645 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2785651 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2785652 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2785655 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2785655 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2785659 INFO  (zkCallback-1429-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2921a5c4 name:ZooKeeperConnection Watcher:127.0.0.1:40584/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2785659 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2785659 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2785661 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2785662 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2785663 INFO  (zkCallback-1430-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5e7034de name:ZooKeeperConnection Watcher:127.0.0.1:40584 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2785664 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2785664 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2785665 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 2785668 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2785674 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2785687 INFO  (zkCallback-1431-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@78341c65 name:ZooKeeperConnection Watcher:127.0.0.1:40584/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2785687 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2785688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2785688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 2785689 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 2785691 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 2785692 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 2785694 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 2785695 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 2785697 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2785697 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2785699 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 2785699 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 2785701 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 2785701 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 2785703 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 2785703 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 2785705 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 2785705 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 2785706 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2785707 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2785709 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 2785709 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 2785711 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 2785712 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 2785714 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 2785714 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 2786030 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1
   [junit4]   2> 2786032 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 2786034 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2080209f{/,null,AVAILABLE}
   [junit4]   2> 2786038 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6e913270{SSL-http/1.1}{127.0.0.1:40598}
   [junit4]   2> 2786039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.e.j.s.Server Started @2793263ms
   [junit4]   2> 2786039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores, hostPort=40598, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/tempDir-001/control/data, hostContext=/}
   [junit4]   2> 2786040 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@1fe91485
   [junit4]   2> 2786040 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/'
   [junit4]   2> 2786079 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2786080 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2786081 INFO  (zkCallback-1432-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6a3731a6 name:ZooKeeperConnection Watcher:127.0.0.1:40584/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2786081 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2786082 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2786083 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2786083 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/solr.xml
   [junit4]   2> 2786099 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores
   [junit4]   2> 2786100 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.CoreContainer New CoreContainer 1133113664
   [junit4]   2> 2786100 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/]
   [junit4]   2> 2786100 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/lib
   [junit4]   2> 2786100 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/lib).
   [junit4]   2> 2786110 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 2786119 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2786121 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2786121 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2786121 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40584/solr
   [junit4]   2> 2786121 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2786121 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2786122 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2786125 INFO  (zkCallback-1434-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3230785b name:ZooKeeperConnection Watcher:127.0.0.1:40584 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2786126 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2786126 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2786128 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2786131 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@79e04c35 name:ZooKeeperConnection Watcher:127.0.0.1:40584/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2786131 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2786132 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 2786134 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 2786136 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 2786138 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 2786140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 2786143 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 2786145 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 2786146 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 2786147 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 2786148 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40598_
   [junit4]   2> 2786149 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:40598_
   [junit4]   2> 2786150 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 2786152 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 2786154 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2786155 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94368158350573575-127.0.0.1:40598_-n_0000000000
   [junit4]   2> 2786155 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40598_
   [junit4]   2> 2786156 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 2786157 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.Overseer Overseer (id=94368158350573575-127.0.0.1:40598_-n_0000000000) starting
   [junit4]   2> 2786159 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 2786165 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2786165 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2786165 INFO  (OverseerCollectionProcessor-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.OverseerCollectionProcessor Process current queue of collection creations
   [junit4]   2> 2786166 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 2786177 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 2786177 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2786178 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores
   [junit4]   2> 2786179 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.SolrCore Created CoreDescriptor: {transient=false, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1, configSetProperties=configsetprops.json, coreNodeName=, config=solrconfig.xml, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/, name=collection1, collection=control_collection, shard=, dataDir=data/, schema=schema.xml}
   [junit4]   2> 2786180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/
   [junit4]   2> 2786180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 2786181 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2
   [junit4]   2> 2786181 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:40598_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 2786183 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 2786183 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 2786183 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 2786183 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2786186 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"https://127.0.0.1:40598",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:40598_",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "roles":null} current state version: 0
   [junit4]   2> 2786187 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"https://127.0.0.1:40598",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:40598_",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "roles":null}
   [junit4]   2> 2786187 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 2786187 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2786189 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2786190 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 1
   [junit4]   2> 2787184 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2787184 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 2787185 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 2787185 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 2787185 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2787186 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/'
   [junit4]   2> 2787203 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 2787209 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 2787227 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.3.0
   [junit4]   2> 2787258 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2787259 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 2787265 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2787536 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2787543 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 2787545 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 2787555 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 2787558 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 2787561 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2787562 WARN  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 2787562 WARN  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 2787562 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2787563 WARN  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 2787563 WARN  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 2787564 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2
   [junit4]   2> 2787565 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2787565 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2787565 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 2787565 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@71f7162
   [junit4]   2> 2787566 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/data
   [junit4]   2> 2787566 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/data/index/
   [junit4]   2> 2787566 WARN  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2787567 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/data/index
   [junit4]   2> 2787567 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@318689d9
   [junit4]   2> 2787568 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@77c41ef5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6ccca440),segFN=segments_1,generation=1}
   [junit4]   2> 2787568 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 2787568 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/data/
   [junit4]   2> 2787568 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 2787573 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2787573 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2787574 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2787574 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2787574 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2787574 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2787574 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2787575 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2787575 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2787578 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2787580 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2787581 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2787582 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2787586 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/ping,/admin/file,/admin/mbeans,standard,/config,/update,/admin/plugins,/schema,/update/json,/admin/segments,/admin/properties,/admin/threads,/update/json/docs,/admin/system,/get,/admin/logging,/replication,/admin/luke,/update/csv
   [junit4]   2> 2787588 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 2787588 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2787588 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2787589 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 2787589 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 2787590 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=41, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2787590 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@77c41ef5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6ccca440),segFN=segments_1,generation=1}
   [junit4]   2> 2787591 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 2787591 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@7de61768[collection1] main
   [junit4]   2> 2787591 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 2787592 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2787592 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2787593 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2787593 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 2787593 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2787593 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2787594 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2787594 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 2787594 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2787595 INFO  (searcherExecutor-4469-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7de61768[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2787596 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 2787596 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 2787596 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 2787596 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1509890535809089536
   [junit4]   2> 2787599 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 3 ms to seed version buckets with highest version 1509890535809089536
   [junit4]   2> 2787599 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 2787600 INFO  (coreLoadExecutor-4468-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 2787600 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:https://127.0.0.1:40598 collection:control_collection shard:shard1
   [junit4]   2> 2787602 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2787605 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94368158350573575-core_node1-n_0000000000
   [junit4]   2> 2787606 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 2787608 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2787608 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2787608 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2787608 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 2787608 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:40598/collection1/
   [junit4]   2> 2787609 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2787609 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:40598/collection1/ has no replicas
   [junit4]   2> 2787609 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:40598/collection1/ shard1
   [junit4]   2> 2787609 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2787614 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2787615 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"https://127.0.0.1:40598",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 2787686 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2787686 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2787689 INFO  (zkCallback-1437-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3b4a6507 name:ZooKeeperConnection Watcher:127.0.0.1:40584/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2787689 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2787690 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2787690 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2787692 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2787693 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 2787693 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2787693 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2787695 INFO  (zkCallback-1438-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@48598593 name:ZooKeeperConnection Watcher:127.0.0.1:40584/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2787695 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2787695 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2787697 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2787698 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 1
   [junit4]   2> 2787698 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 2787702 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2787703 INFO  (zkCallback-1437-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2787705 INFO  (zkCallback-1437-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 2
   [junit4]   2> 2787705 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 2
   [junit4]   2> 2787716 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are https://127.0.0.1:40598/collection1/ and leader is https://127.0.0.1:40598/collection1/
   [junit4]   2> 2787716 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:40598
   [junit4]   2> 2787716 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2787716 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 2787716 INFO  (coreZkRegister-4462-thread-1-processing-r:core_node1 x:collection1 n:127.0.0.1:40598_ s:shard1 c:control_collection) [n:127.0.0.1:40598_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 2787718 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2787718 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "base_url":"https://127.0.0.1:40598",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:40598_",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "roles":null} current state version: 2
   [junit4]   2> 2787720 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "base_url":"https://127.0.0.1:40598",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:40598_",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "roles":null}
   [junit4]   2> 2787720 INFO  (OverseerStateUpdate-94368158350573575-127.0.0.1:40598_-n_0000000000) [n:127.0.0.1:40598_    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 2787721 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2787721 INFO  (zkCallback-1437-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2787723 INFO  (zkCallback-1437-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 2787723 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 2787826 INFO  (zkCallback-1437-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2787826 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2787828 INFO  (zkCallback-1437-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 2787829 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:40598_) [n:127.0.0.1:40598_    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 2788074 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/cores/collection1
   [junit4]   2> 2788075 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001
   [junit4]   2> 2788076 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 2788083 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1c713cbb{/,null,AVAILABLE}
   [junit4]   2> 2788086 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.e.j.s.ServerConnector Started ServerConnector@666c4c0c{SSL-http/1.1}{127.0.0.1:48730}
   [junit4]   2> 2788086 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.e.j.s.Server Started @2795310ms
   [junit4]   2> 2788086 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/tempDir-001/jetty1, hostPort=48730, hostContext=/, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/cores, solrconfig=solrconfig.xml}
   [junit4]   2> 2788086 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@1fe91485
   [junit4]   2> 2788087 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/'
   [junit4]   2> 2788126 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2788130 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2788130 INFO  (zkCallback-1439-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@d5b41de name:ZooKeeperConnection Watcher:127.0.0.1:40584/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2788131 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2788131 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2788132 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2788133 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/solr.xml
   [junit4]   2> 2788161 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/cores
   [junit4]   2> 2788161 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.CoreContainer New CoreContainer 2100620127
   [junit4]   2> 2788162 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/]
   [junit4]   2> 2788162 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/lib
   [junit4]   2> 2788162 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/lib).
   [junit4]   2> 2788176 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 2788180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2788182 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2788183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2788183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40584/solr
   [junit4]   2> 2788183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2788183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2788184 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2788186 INFO  (zkCallback-1441-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@78e3d78b name:ZooKeeperConnection Watcher:127.0.0.1:40584 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2788188 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2788188 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2788191 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2788195 INFO  (zkCallback-1442-thread-1-processing-n:127.0.0.1:48730_) [n:127.0.0.1:48730_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@40257e2 name:ZooKeeperConnection Watcher:127.0.0.1:40584/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2788196 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2788202 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2789207 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48730_
   [junit4]   2> 2789208 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:48730_
   [junit4]   2> 2789211 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2789211 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94368158350573580-127.0.0.1:48730_-n_0000000001
   [junit4]   2> 2789212 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94368158350573575-127.0.0.1:40598_-n_0000000000 to know if I could be the leader
   [junit4]   2> 2789221 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 2789221 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2789222 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/cores
   [junit4]   2> 2789223 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6B115E9977A97773]) [n:127.0.0.1:48730_    ] o.a.s.c.SolrCore Created CoreDescriptor: {collection=collection1, coreNodeName=, dataDir=data/, configSetProperties=configsetprops.json, name=collection1, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/cores/collection1/, shard=, config=solrconfig.xml, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6B115E9977A97773-001/shard-1-001/cores/collection1, schema=schema.xml, loadOnStartup=true, t

[...truncated too long message...]

nServerInterceptor.java:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> 	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:227)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:197)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:50)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 53 more
   [junit4]   2> 5159347 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[6B115E9977A97773]-worker) [n:127.0.0.1:51499_t_ek%2Foe c:addReplicaColl s:shard2 r:core_node6 x:addReplicaColl_shard2_replica3] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Aug 18, 2015 9:03:20 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {multiDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_i1=FST50, timestamp=FST50, text=PostingsFormat(name=Direct), range_facet_l_dv=PostingsFormat(name=Memory doPackFST= true), range_facet_i_dv=FST50, a_t=FST50, _version_=FST50, other_tl1=FST50, id=PostingsFormat(name=Memory doPackFST= true), range_facet_l=FST50, intDefault=FST50}, docValues:{timestamp=DocValuesFormat(name=Asserting), range_facet_l_dv=DocValuesFormat(name=Lucene50), range_facet_i_dv=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=el_GR, timezone=Australia/West
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=2,free=299213632,total=534249472
   [junit4]   2> NOTE: All tests run in this JVM: [TestWordDelimiterFilterFactory, TestStressUserVersions, TestNamedUpdateProcessors, SolrIndexConfigTest, TestConfig, JavabinLoaderTest, MBeansHandlerTest, TestReversedWildcardFilterFactory, TestExactStatsCache, UUIDUpdateProcessorFallbackTest, TestCryptoKeys, TestDocBasedVersionConstraints, AnalysisErrorHandlingTest, TestConfigOverlay, TestObjectReleaseTracker, TestCollectionAPI, SuggesterFSTTest, TestCustomSort, ParsingFieldUpdateProcessorsTest, TestManagedSchemaFieldResource, TestRemoteStreaming, TestCloudManagedSchemaConcurrent, TestBinaryField, TestCloudInspectUtil, CleanupOldIndexTest, ExternalFileFieldSortTest, TestStressLucene, TestCodecSupport, PeerSyncTest, ConnectionManagerTest, UUIDFieldTest, DistributedFacetPivotSmallAdvancedTest, TestFieldResource, TestSolrConfigHandler, StressHdfsTest, TestSweetSpotSimilarityFactory, DefaultValueUpdateProcessorTest, RequestHandlersTest, TestReplicationHandler, TestRealTimeGet, TestRandomFaceting, ZkNodePropsTest, DebugComponentTest, BasicDistributedZkTest, TermVectorComponentTest, TestMiniSolrCloudClusterSSL, IndexBasedSpellCheckerTest, TestChildDocTransformer, TestJmxMonitoredMap, StandardRequestHandlerTest, TestBadConfig, TestCursorMarkWithoutUniqueKey, TestFieldSortValues, HdfsDirectoryTest, MultiTermTest, TestQuerySenderListener, HdfsRecoverLeaseTest, ExitableDirectoryReaderTest, IndexSchemaTest, HighlighterTest, HLLUtilTest, TestSolrConfigHandlerCloud, HdfsSyncSliceTest, CoreAdminRequestStatusTest, TestUniqueKeyFieldResource, TestMergePolicyConfig, UpdateRequestProcessorFactoryTest, TestTolerantSearch, MultiThreadedOCPTest, SolrCloudExampleTest, TestLazyCores, AddBlockUpdateTest, SchemaVersionSpecificBehaviorTest, UniqFieldsUpdateProcessorFactoryTest, SaslZkACLProviderTest, SimplePostToolTest, LeaderInitiatedRecoveryOnCommitTest, CoreMergeIndexesAdminHandlerTest, BitVectorTest, MigrateRouteKeyTest, DateFieldTest, TestArbitraryIndexDir, TestSolrIndexConfig, TestFaceting, DeleteLastCustomShardedReplicaTest, SyncSliceTest, OpenCloseCoreStressTest, ClusterStateUpdateTest, LeaderElectionTest, ZkSolrClientTest, TestDistributedSearch, PreAnalyzedFieldTest, TestSystemIdResolver, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, TestUtils, TestDocumentBuilder, FileUtilsTest, CircularListTest, DistributedMLTComponentTest, TestRTGBase, DistributedIntervalFacetingTest, TestJettySolrRunner, AssignTest, CloudExitableDirectoryReaderTest, CollectionsAPIAsyncDistributedZkTest, DeleteInactiveReplicaTest, DistribJoinFromCollectionTest, OverseerStatusTest, RecoveryAfterSoftCommitTest, TestDownShardTolerantSearch, TestRebalanceLeaders, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [464/531] on J0 in 516.66s, 1 test, 1 error <<< FAILURES!

[...truncated 223 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:733: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build.xml:230: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/common-build.xml:524: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1449: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1003: There were test failures: 531 suites, 2104 tests, 2 errors, 26 ignored (14 assumptions)

Total time: 262 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.3 - Build # 8 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.3/8/

7 tests failed.
REGRESSION:  org.apache.solr.TestDistributedSearch.test

Error Message:
Captured an uncaught exception in thread: Thread[id=74293, name=Thread-69129, state=RUNNABLE, group=TGRP-TestDistributedSearch]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=74293, name=Thread-69129, state=RUNNABLE, group=TGRP-TestDistributedSearch]
	at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869:84B669169D36A591]:0)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:36086/dau/g/collection1: java.lang.NullPointerException
	at org.apache.solr.search.grouping.distributed.responseprocessor.TopGroupsShardResponseProcessor.process(TopGroupsShardResponseProcessor.java:102)
	at org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:748)
	at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:731)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:410)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2068)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:669)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:462)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:106)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

	at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
	at org.apache.solr.TestDistributedSearch$1.run(TestDistributedSearch.java:1110)


REGRESSION:  org.apache.solr.cloud.AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869:56E333201DF045B0]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse(AsyncCallRequestStatusResponseTest.java:38)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=11397, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11397, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:58229, http://127.0.0.1:55256, http://127.0.0.1:55659, http://127.0.0.1:36930, http://127.0.0.1:49275]
	at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869]:0)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:897)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:58229, http://127.0.0.1:55256, http://127.0.0.1:55659, http://127.0.0.1:36930, http://127.0.0.1:49275]
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:58229: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qn-
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	... 5 more


REGRESSION:  org.apache.solr.cloud.DistribJoinFromCollectionTest.test

Error Message:
KeeperErrorCode = Session expired for /live_nodes

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /live_nodes
	at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869:84B669169D36A591]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:325)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:325)
	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:561)
	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:239)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1807)
	at org.apache.solr.cloud.DistribJoinFromCollectionTest.test(DistribJoinFromCollectionTest.java:95)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:57397/vo_v/collection1: ClusterState says we are the leader (http://127.0.0.1:57397/vo_v/collection1), but locally we don't think so. Request came from null

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:57397/vo_v/collection1: ClusterState says we are the leader (http://127.0.0.1:57397/vo_v/collection1), but locally we don't think so. Request came from null
	at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869:84B669169D36A591]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
	at org.apache.solr.BaseDistributedSearchTestCase.add(BaseDistributedSearchTestCase.java:511)
	at org.apache.solr.cloud.BasicDistributedZkTest.testUpdateProcessorsRunOnlyOnce(BasicDistributedZkTest.java:639)
	at org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:350)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
shard3 is not consistent.  Got 247 from https://127.0.0.1:56737/_zim/hi/collection1lastClient and got 164 from https://127.0.0.1:53641/_zim/hi/collection1

Stack Trace:
java.lang.AssertionError: shard3 is not consistent.  Got 247 from https://127.0.0.1:56737/_zim/hi/collection1lastClient and got 164 from https://127.0.0.1:53641/_zim/hi/collection1
	at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869:84B669169D36A591]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1244)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1223)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:165)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=79096, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=79096, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:48997: Could not find collection : awholynewstresscollection_collection2_3
	at __randomizedtesting.SeedInfo.seed([CE256CC33CAC869]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)




Build Log:
[...truncated 10509 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/init-core-data-001
   [junit4]   2> 556635 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 556635 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_zim/hi
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 556674 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 556683 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 556688 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 556704 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_52557_hdfs____p1gy2s/webapp
   [junit4]   2> 556829 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 557314 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52557
   [junit4]   2> 557476 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 557477 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 557493 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_53677_datanode____.3qnrvl/webapp
   [junit4]   2> 557691 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 558153 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53677
   [junit4]   2> 558187 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 558189 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 558205 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_51691_datanode____pjddp5/webapp
   [junit4]   2> 558252 INFO  (IPC Server handler 3 on 52095) [    ] BlockStateChange BLOCK* processReport: from storage DS-06bd408f-01e8-4260-a36a-1ab09003185d node DatanodeRegistration(127.0.0.1, datanodeUuid=8e1655f6-c766-4907-b501-b11a31068378, infoPort=53677, ipcPort=44740, storageInfo=lv=-56;cid=testClusterID;nsid=2032221109;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 558253 INFO  (IPC Server handler 3 on 52095) [    ] BlockStateChange BLOCK* processReport: from storage DS-3b9c02a3-3c16-44ce-9284-15b17e4693ac node DatanodeRegistration(127.0.0.1, datanodeUuid=8e1655f6-c766-4907-b501-b11a31068378, infoPort=53677, ipcPort=44740, storageInfo=lv=-56;cid=testClusterID;nsid=2032221109;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 558349 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 558860 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[CE256CC33CAC869]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51691
   [junit4]   2> 558949 INFO  (IPC Server handler 4 on 52095) [    ] BlockStateChange BLOCK* processReport: from storage DS-be9f15fb-0c1a-41e5-ae5d-2ce9c8e8e5dd node DatanodeRegistration(127.0.0.1, datanodeUuid=8e5ecb1a-4c78-4b3d-9158-1f33fa781b8d, infoPort=51691, ipcPort=55865, storageInfo=lv=-56;cid=testClusterID;nsid=2032221109;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 558950 INFO  (IPC Server handler 4 on 52095) [    ] BlockStateChange BLOCK* processReport: from storage DS-108c8116-2f3d-4ce1-9765-c5e8d338dd30 node DatanodeRegistration(127.0.0.1, datanodeUuid=8e5ecb1a-4c78-4b3d-9158-1f33fa781b8d, infoPort=51691, ipcPort=55865, storageInfo=lv=-56;cid=testClusterID;nsid=2032221109;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 559025 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 559025 INFO  (Thread-15766) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 559025 INFO  (Thread-15766) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 559125 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.ZkTestServer start zk server on port:51301
   [junit4]   2> 559127 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559128 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559132 INFO  (zkCallback-193-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@703bd33f name:ZooKeeperConnection Watcher:127.0.0.1:51301 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559132 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559132 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559132 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 559137 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559138 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559139 INFO  (zkCallback-194-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@27a0d7f6 name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559139 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559139 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559139 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 559144 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 559146 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 559149 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 559151 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 559151 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 559154 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 559155 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 559157 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 559157 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 559159 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 559159 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 559161 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 559161 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 559163 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 559163 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 559166 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 559166 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 559170 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 559170 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 559173 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 559173 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 559174 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 559175 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 559179 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 559179 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 559183 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559184 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559185 INFO  (zkCallback-195-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5b761d5a name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559185 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559186 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559485 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1
   [junit4]   2> 559487 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 559489 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7ddcc13e{/_zim/hi,null,AVAILABLE}
   [junit4]   2> 559494 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.e.j.s.ServerConnector Started ServerConnector@7afc368d{SSL-http/1.1}{127.0.0.1:46910}
   [junit4]   2> 559494 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.e.j.s.Server Started @562692ms
   [junit4]   2> 559495 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/_zim/hi, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores, solr.data.dir=hdfs://localhost:52095/hdfs__localhost_52095__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.3_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001_tempDir-002_control_data, hostPort=46910}
   [junit4]   2> 559495 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@1fe91485
   [junit4]   2> 559495 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/'
   [junit4]   2> 559512 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559513 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559516 INFO  (zkCallback-196-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@453510db name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559516 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559516 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559519 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 559520 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/solr.xml
   [junit4]   2> 559536 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores
   [junit4]   2> 559536 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.CoreContainer New CoreContainer 224294256
   [junit4]   2> 559536 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/]
   [junit4]   2> 559536 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/lib
   [junit4]   2> 559536 WARN  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/lib).
   [junit4]   2> 559545 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 559548 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 559550 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 559550 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 559550 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51301/solr
   [junit4]   2> 559550 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 559550 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 559552 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559553 INFO  (zkCallback-198-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5a8b238a name:ZooKeeperConnection Watcher:127.0.0.1:51301 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559554 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559554 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 559558 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 559559 INFO  (zkCallback-199-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@613e9592 name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 559559 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 559560 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 559563 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 559566 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 559568 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 559570 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 559572 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 559574 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 559576 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 559577 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 559578 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46910__zim%2Fhi
   [junit4]   2> 559578 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:46910__zim%2Fhi
   [junit4]   2> 559580 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 559581 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 559583 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 559583 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000
   [junit4]   2> 559584 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46910__zim%2Fhi
   [junit4]   2> 559584 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 559585 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.Overseer Overseer (id=94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) starting
   [junit4]   2> 559587 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 559593 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 559593 INFO  (OverseerCollectionProcessor-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.OverseerCollectionProcessor Process current queue of collection creations
   [junit4]   2> 559594 INFO  (OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 559594 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 559615 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 559615 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 559616 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores
   [junit4]   2> 559617 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.SolrCore Created CoreDescriptor: {loadOnStartup=true, collection=control_collection, config=solrconfig.xml, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1, schema=schema.xml, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1/, shard=, coreNodeName=, name=collection1, transient=false, dataDir=data/, configSetProperties=configsetprops.json}
   [junit4]   2> 559617 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1/
   [junit4]   2> 559617 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 559619 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0
   [junit4]   2> 559619 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 559621 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 559621 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 559622 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 559622 INFO  (zkCallback-199-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 559623 INFO  (OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "base_url":"https://127.0.0.1:46910/_zim/hi",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "state":"down",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "node_name":"127.0.0.1:46910__zim%2Fhi",
   [junit4]   2>   "roles":null} current state version: 0
   [junit4]   2> 559624 INFO  (OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "base_url":"https://127.0.0.1:46910/_zim/hi",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "state":"down",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "node_name":"127.0.0.1:46910__zim%2Fhi",
   [junit4]   2>   "roles":null}
   [junit4]   2> 559624 INFO  (OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 559624 INFO  (OverseerStateUpdate-94364008793571333-127.0.0.1:46910__zim%2Fhi-n_0000000000) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 559625 INFO  (zkCallback-199-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 559628 INFO  (zkCallback-199-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 1
   [junit4]   2> 560622 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 560622 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 560623 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 560623 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 560624 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 560624 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1/'
   [junit4]   2> 560642 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 560648 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 560661 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.3.0
   [junit4]   2> 560697 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 560698 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 560705 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 560933 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 560934 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 560936 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 560939 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 560964 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0
   [junit4]   2> 560965 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 560965 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 560965 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:52095/solr_hdfs_home
   [junit4]   2> 560965 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 560965 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_CE256CC33CAC869-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 560966 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@23a70112
   [junit4]   2> 560966 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 560984 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data for locality metrics.
   [junit4]   2> 560984 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 560985 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 560996 WARN  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 560996 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 561005 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 561005 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 561005 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 561013 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 561013 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index for locality metrics.
   [junit4]   2> 561013 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 561013 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5d0041cd
   [junit4]   2> 561050 INFO  (IPC Server handler 8 on 52095) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54084 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-06bd408f-01e8-4260-a36a-1ab09003185d:NORMAL:127.0.0.1:37256|RBW], ReplicaUnderConstruction[[DISK]DS-108c8116-2f3d-4ce1-9765-c5e8d338dd30:NORMAL:127.0.0.1:54084|FINALIZED]]} size 0
   [junit4]   2> 561052 INFO  (IPC Server handler 9 on 52095) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37256 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-108c8116-2f3d-4ce1-9765-c5e8d338dd30:NORMAL:127.0.0.1:54084|FINALIZED], ReplicaUnderConstruction[[DISK]DS-3b9c02a3-3c16-44ce-9284-15b17e4693ac:NORMAL:127.0.0.1:37256|FINALIZED]]} size 0
   [junit4]   2> 561055 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b8369659 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@381847d9); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 561055 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 561057 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:52095/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 561061 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 561061 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 561063 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 561063 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 561063 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 561063 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 561064 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 561067 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 561068 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 561073 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 561074 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 561076 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 561077 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 561081 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /update/json/docs,standard,/admin/logging,/replication,/schema,/config,/admin/luke,/admin/ping,/update/csv,/update/json,/update,/admin/plugins,/admin/properties,/get,/admin/segments,/admin/system,/admin/mbeans,/admin/threads,/admin/file
   [junit4]   2> 561082 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 561083 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 561083 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing HdfsUpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 tlogDfsReplication=2
   [junit4]   2> 561095 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 561095 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 561097 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=23, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 561105 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@b8369659 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@381847d9); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 561105 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 561107 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@1a2fb6b6[collection1] main
   [junit4]   2> 561107 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 561109 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 561109 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 561110 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 561110 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 561110 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 561111 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 561111 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 561111 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 561111 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 561113 INFO  (searcherExecutor-1324-thread-1-processing-x:collection1 s:shard1 r:core_node1 c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1a2fb6b6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 561115 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 561115 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version buckets from index
   [junit4]   2> 561115 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 561115 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1509824142886567936
   [junit4]   2> 561115 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 0 ms to seed version buckets with highest version 1509824142886567936
   [junit4]   2> 561115 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 561116 INFO  (coreLoadExecutor-1323-thread-1-processing-n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 561120 INFO  (coreZkRegister-1317-thread-1-processing-x:collection1 s:shard1 r:core_node1 c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:https://127.0.0.1:46910/_zim/hi collection:control_collection shard:shard1
   [junit4]   2> 561123 INFO  (coreZkRegister-1317-thread-1-processing-x:collection1 s:shard1 r:core_node1 c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 561125 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 561125 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 561128 INFO  (zkCallback-201-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46f66847 name:ZooKeeperConnection Watcher:127.0.0.1:51301/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 561128 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 561128 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 561128 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[CE256CC33CAC869]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 561134 INFO  (coreZkRegister-1317-thread-1-processing-x:collection1 s:shard1 r:core_node1 c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94364008793571333-core_node1-n_0000000000
   [junit4]   2> 561135 INFO  (coreZkRegister-1317-thread-1-processing-x:collection1 s:shard1 r:core_node1 c:control_collection n:127.0.0.1:46910__zim%2Fhi) [n:127.0.0.1:46910__zim%2Fhi c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 561137 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-

[...truncated too long message...]

06)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:387)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> 	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:227)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:197)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:50)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 53 more
   [junit4]   2> 2939348 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[CE256CC33CAC869]-worker) [n:127.0.0.1:45483_ c:addReplicaColl s:shard2 r:core_node6 x:addReplicaColl_shard2_replica3] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Aug 18, 2015 3:28:06 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53), sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_CR, timezone=Etc/GMT+2
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=2,free=146510560,total=533200896
   [junit4]   2> NOTE: All tests run in this JVM: [TestSchemaSimilarityResource, TestBinaryField, TestRealTimeGet, TestJoin, BasicZkTest, StressHdfsTest, TestStressLucene, LeaderElectionIntegrationTest, TestSolr4Spatial, TestReversedWildcardFilterFactory, TestRebalanceLeaders, TestSolrDeletionPolicy1, FastVectorHighlighterTest, HdfsDirectoryTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, MultiTermTest, ConcurrentDeleteAndCreateCollectionTest, LeaderInitiatedRecoveryOnCommitTest, TestConfigOverlay, ClusterStateTest, BitVectorTest, TestDocumentBuilder, URLClassifyProcessorTest, TestWordDelimiterFilterFactory, TestPivotHelperCode, ExitableDirectoryReaderTest, HdfsChaosMonkeySafeLeaderTest, UniqFieldsUpdateProcessorFactoryTest, SuggestComponentTest, DistribJoinFromCollectionTest, ChaosMonkeyNothingIsSafeTest, DefaultValueUpdateProcessorTest, SolrRequestParserTest, TestReqParamsAPI, CloudMLTQParserTest, CoreAdminRequestStatusTest, DistributedSpellCheckComponentTest, DistributedFacetPivotSmallAdvancedTest, TestCloudManagedSchemaConcurrent, TestDistributedSearch, BasicDistributedZkTest, TestChildDocTransformer, TestManagedSynonymFilterFactory, TestNamedUpdateProcessors, SuggesterFSTTest, PeerSyncTest, DistanceUnitsTest, CurrencyFieldXmlFileTest, VMParamsZkACLAndCredentialsProvidersTest, TestCursorMarkWithoutUniqueKey, JavabinLoaderTest, AddSchemaFieldsUpdateProcessorFactoryTest, CoreMergeIndexesAdminHandlerTest, TermsComponentTest, BigEndianAscendingWordDeserializerTest, MBeansHandlerTest, CSVRequestHandlerTest, TestRemoteStreaming, TestBadConfig, TestCollectionAPI, TestArbitraryIndexDir, DeleteInactiveReplicaTest, TestRandomFaceting, HdfsDirectoryFactoryTest, TestBlendedInfixSuggestions, SuggesterWFSTTest, TestPhraseSuggestions, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [380/531] on J0 in 440.43s, 1 test, 1 error <<< FAILURES!

[...truncated 515 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:733: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build.xml:230: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/common-build.xml:524: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1449: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1003: There were test failures: 531 suites, 2104 tests, 5 errors, 2 failures, 26 ignored (14 assumptions)

Total time: 189 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.3 - Build # 7 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.3/7/

4 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=53555, name=collection3, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=53555, name=collection3, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:44540/k_t/yn: Could not find collection : awholynewstresscollection_collection3_0
	at __randomizedtesting.SeedInfo.seed([1FB1272CD8B4CB56]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1085)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:894)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestDynamicLoading

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([1FB1272CD8B4CB56]:0)
	at org.apache.lucene.util.TestRuleRestoreSystemProperties.before(TestRuleRestoreSystemProperties.java:53)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.BadComponentTest

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.DocValuesMultiTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([1FB1272CD8B4CB56]:0)




Build Log:
[...truncated 10660 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/init-core-data-001
   [junit4]   2> 948918 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 948918 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /k_t/yn
   [junit4]   2> 949572 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 950341 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 950660 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 950687 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 950791 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 950891 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_49145_hdfs____dnfvxz/webapp
   [junit4]   2> 951195 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 952320 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49145
   [junit4]   2> 953186 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 953197 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 953210 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_42276_datanode____ti93ui/webapp
   [junit4]   2> 953379 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 954068 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42276
   [junit4]   2> 954194 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 954196 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 954211 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_47795_datanode____t1tonr/webapp
   [junit4]   2> 954345 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 954976 INFO  (IPC Server handler 2 on 49181) [    ] BlockStateChange BLOCK* processReport: from storage DS-8ab9bdc3-21c6-439a-82aa-7388b325ce18 node DatanodeRegistration(127.0.0.1, datanodeUuid=b8676401-da84-4f26-8595-49397b842261, infoPort=42276, ipcPort=56150, storageInfo=lv=-56;cid=testClusterID;nsid=1475220372;c=0), blocks: 0, hasStaleStorages: true, processing time: 3 msecs
   [junit4]   2> 954979 INFO  (IPC Server handler 2 on 49181) [    ] BlockStateChange BLOCK* processReport: from storage DS-1ae4ae06-1d81-4006-becb-e3ff05b9bac7 node DatanodeRegistration(127.0.0.1, datanodeUuid=b8676401-da84-4f26-8595-49397b842261, infoPort=42276, ipcPort=56150, storageInfo=lv=-56;cid=testClusterID;nsid=1475220372;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 955110 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1FB1272CD8B4CB56]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47795
   [junit4]   2> 955253 INFO  (IPC Server handler 2 on 49181) [    ] BlockStateChange BLOCK* processReport: from storage DS-18cf5bc6-e992-4175-95e1-43e67a839be1 node DatanodeRegistration(127.0.0.1, datanodeUuid=50c0256a-3db2-4308-a7eb-9102b2979ac9, infoPort=47795, ipcPort=49913, storageInfo=lv=-56;cid=testClusterID;nsid=1475220372;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 955254 INFO  (IPC Server handler 2 on 49181) [    ] BlockStateChange BLOCK* processReport: from storage DS-85b59010-3641-400d-925b-4bed344c103a node DatanodeRegistration(127.0.0.1, datanodeUuid=50c0256a-3db2-4308-a7eb-9102b2979ac9, infoPort=47795, ipcPort=49913, storageInfo=lv=-56;cid=testClusterID;nsid=1475220372;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 956002 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 956003 INFO  (Thread-49150) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 956003 INFO  (Thread-49150) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 956103 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.ZkTestServer start zk server on port:56104
   [junit4]   2> 956103 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 956117 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 956124 INFO  (zkCallback-1319-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46539568 name:ZooKeeperConnection Watcher:127.0.0.1:56104 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 956124 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 956125 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 956125 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 956132 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 956134 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 956135 INFO  (zkCallback-1320-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7766ec35 name:ZooKeeperConnection Watcher:127.0.0.1:56104/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 956135 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 956135 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 956136 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 956138 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 956140 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 956141 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 956143 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 956143 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 956147 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 956148 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 956151 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 956151 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 956154 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 956154 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 956157 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 956157 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 956160 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 956160 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 956163 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 956163 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 956166 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 956167 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 956170 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 956170 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 956173 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 956173 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 956176 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 956176 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 956536 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/cores/collection1
   [junit4]   2> 956538 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 956541 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4b26338d{/k_t/yn,null,AVAILABLE}
   [junit4]   2> 956542 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.e.j.s.ServerConnector Started ServerConnector@39029486{HTTP/1.1}{127.0.0.1:52030}
   [junit4]   2> 956542 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.e.j.s.Server Started @959597ms
   [junit4]   2> 956542 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/k_t/yn, solr.data.dir=hdfs://localhost:49181/hdfs__localhost_49181__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.3_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001_tempDir-002_control_data, hostPort=52030, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/cores}
   [junit4]   2> 956542 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@1fe91485
   [junit4]   2> 956542 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/'
   [junit4]   2> 956559 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 956561 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 956562 INFO  (zkCallback-1321-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4aaa3915 name:ZooKeeperConnection Watcher:127.0.0.1:56104/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 956562 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 956562 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 956564 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 956564 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/solr.xml
   [junit4]   2> 956580 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/cores
   [junit4]   2> 956580 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.CoreContainer New CoreContainer 1955343669
   [junit4]   2> 956580 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/]
   [junit4]   2> 956580 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/lib
   [junit4]   2> 956580 WARN  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/lib).
   [junit4]   2> 956588 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 956590 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 956590 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 956591 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 956591 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56104/solr
   [junit4]   2> 956591 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 956591 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 956592 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 956594 INFO  (zkCallback-1323-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@63d4890a name:ZooKeeperConnection Watcher:127.0.0.1:56104 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 956595 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 956595 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 956598 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 956599 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@10abcaa6 name:ZooKeeperConnection Watcher:127.0.0.1:56104/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 956599 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 956601 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 956603 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 956605 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 956608 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 956610 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 956613 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 956615 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 956617 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 956618 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 956619 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52030_k_t%2Fyn
   [junit4]   2> 956620 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:52030_k_t%2Fyn
   [junit4]   2> 956621 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 956623 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 956624 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 956625 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000
   [junit4]   2> 956626 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52030_k_t%2Fyn
   [junit4]   2> 956626 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 956627 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.Overseer Overseer (id=94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) starting
   [junit4]   2> 956629 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 956636 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 956636 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 956636 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 956637 INFO  (OverseerCollectionProcessor-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.OverseerCollectionProcessor Process current queue of collection creations
   [junit4]   2> 956651 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 956651 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 956652 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/cores
   [junit4]   2> 956653 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.SolrCore Created CoreDescriptor: {shard=, config=solrconfig.xml, coreNodeName=, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/cores/collection1, configSetProperties=configsetprops.json, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/cores/collection1/, loadOnStartup=true, collection=control_collection, transient=false, dataDir=data/, name=collection1, schema=schema.xml}
   [junit4]   2> 956653 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/cores/collection1/
   [junit4]   2> 956653 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 956655 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 956655 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0
   [junit4]   2> 956655 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 956655 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 956656 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 956656 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 956657 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:52030_k_t%2Fyn",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "base_url":"http://127.0.0.1:52030/k_t/yn",
   [junit4]   2>   "shard":null} current state version: 0
   [junit4]   2> 956657 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:52030_k_t%2Fyn",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "base_url":"http://127.0.0.1:52030/k_t/yn",
   [junit4]   2>   "shard":null}
   [junit4]   2> 956657 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 956658 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 956659 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 956660 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 1
   [junit4]   2> 957656 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 957657 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 957657 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 957657 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 957658 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 957658 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/cores/collection1/'
   [junit4]   2> 957696 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 957705 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 957719 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.3.0
   [junit4]   2> 957748 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 957750 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 957770 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 958046 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 958055 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 958058 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 958070 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 958074 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 958078 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 958079 WARN  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 958079 WARN  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 958079 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 958080 WARN  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 958080 WARN  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 958082 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0
   [junit4]   2> 958082 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 958082 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 958094 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49181/solr_hdfs_home
   [junit4]   2> 958094 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 958094 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 958095 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@19f2d9af
   [junit4]   2> 958095 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49181/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 958158 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:49181/solr_hdfs_home/control_collection/core_node1/data for locality metrics.
   [junit4]   2> 958158 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49181/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 958164 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:49181/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 958185 WARN  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:49181/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 958185 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49181/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 958204 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsLocalityReporter Registering direcotry hdfs://localhost:49181/solr_hdfs_home/control_collection/core_node1/data/index for locality metrics.
   [junit4]   2> 958204 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49181/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 958204 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@71a8b41c
   [junit4]   2> 958496 INFO  (IPC Server handler 6 on 49181) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54108 is added to blk_1073741825_1001{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-8ab9bdc3-21c6-439a-82aa-7388b325ce18:NORMAL:127.0.0.1:54108|RBW], ReplicaUnderConstruction[[DISK]DS-85b59010-3641-400d-925b-4bed344c103a:NORMAL:127.0.0.1:51481|RBW]]} size 71
   [junit4]   2> 958503 INFO  (IPC Server handler 7 on 49181) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51481 is added to blk_1073741825_1001 size 71
   [junit4]   2> 958902 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@b7dfed3f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50a28b65; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 958902 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 958905 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:49181/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 958909 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 958909 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 958910 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 958910 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 958910 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 958910 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 958911 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 958911 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 958911 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 958914 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 958916 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 958917 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 958919 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 958931 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /get,/admin/system,/update/json/docs,/admin/threads,/admin/luke,/replication,/admin/file,/admin/plugins,/admin/properties,/admin/segments,standard,/update/csv,/update,/admin/ping,/schema,/config,/update/json,/admin/mbeans,/admin/logging
   [junit4]   2> 958933 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 958937 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 958937 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing HdfsUpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 tlogDfsReplication=2
   [junit4]   2> 958960 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 958960 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 958962 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=85.2431640625, floorSegmentMB=0.861328125, forceMergeDeletesPctAllowed=6.460220982143582, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 959069 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@b7dfed3f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50a28b65; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 959069 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 959069 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@35ce0533[collection1] main
   [junit4]   2> 959070 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 959071 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 959071 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 959072 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 959072 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 959072 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 959072 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 959073 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 959073 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 959073 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 959075 INFO  (searcherExecutor-1307-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@35ce0533[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 959083 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 959084 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version buckets from index
   [junit4]   2> 959084 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 959085 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1509692274745802752
   [junit4]   2> 959085 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1 ms to seed version buckets with highest version 1509692274745802752
   [junit4]   2> 959085 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 959087 INFO  (coreLoadExecutor-1306-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 959087 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:52030/k_t/yn collection:control_collection shard:shard1
   [junit4]   2> 959089 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 959093 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94355766969761796-core_node1-n_0000000000
   [junit4]   2> 959094 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 959096 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 959096 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 959096 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 959096 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 959098 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 959100 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 2
   [junit4]   2> 959101 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52030/k_t/yn/collection1/
   [junit4]   2> 959101 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 959101 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:52030/k_t/yn/collection1/ has no replicas
   [junit4]   2> 959101 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52030/k_t/yn/collection1/ shard1
   [junit4]   2> 959102 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 959105 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 959106 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:52030/k_t/yn",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 959157 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 959158 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 959160 INFO  (zkCallback-1326-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ba67bc1 name:ZooKeeperConnection Watcher:127.0.0.1:56104/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 959160 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 959160 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 959161 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 959164 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 959164 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 959165 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 959165 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 959167 INFO  (zkCallback-1327-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3e4a5f4d name:ZooKeeperConnection Watcher:127.0.0.1:56104/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 959167 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 959168 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 959170 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 959170 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 2
   [junit4]   2> 959170 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 959171 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 959172 INFO  (zkCallback-1326-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 959176 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 959176 INFO  (zkCallback-1326-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 3
   [junit4]   2> 959210 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:52030/k_t/yn/collection1/ and leader is http://127.0.0.1:52030/k_t/yn/collection1/
   [junit4]   2> 959210 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52030/k_t/yn
   [junit4]   2> 959210 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 959210 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 959210 INFO  (coreZkRegister-1300-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn c:control_collection x:collection1 r:core_node1 s:shard1) [n:127.0.0.1:52030_k_t%2Fyn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 959212 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.DistributedQueue NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 959213 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:52030_k_t%2Fyn",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "base_url":"http://127.0.0.1:52030/k_t/yn",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "shard":"shard1"} current state version: 3
   [junit4]   2> 959214 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:52030_k_t%2Fyn",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "base_url":"http://127.0.0.1:52030/k_t/yn",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "shard":"shard1"}
   [junit4]   2> 959215 INFO  (OverseerStateUpdate-94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 959216 INFO  (zkCallback-1326-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 959217 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 959219 INFO  (zkCallback-1326-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 959222 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 4
   [junit4]   2> 959325 INFO  (zkCallback-1326-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 959325 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 959326 INFO  (zkCallback-1326-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 5
   [junit4]   2> 959327 INFO  (zkCallback-1324-thread-1-processing-n:127.0.0.1:52030_k_t%2Fyn) [n:127.0.0.1:52030_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader Updated cluster state version to 5
   [junit4]   2> 959543 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/cores/collection1
   [junit4]   2> 959544 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001
   [junit4]   2> 959544 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.e.j.s.Server jetty-9.2.11.v20150529
   [junit4]   2> 959547 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@393470ab{/k_t/yn,null,AVAILABLE}
   [junit4]   2> 959547 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.e.j.s.ServerConnector Started ServerConnector@225d1b14{HTTP/1.1}{127.0.0.1:44540}
   [junit4]   2> 959547 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.e.j.s.Server Started @962602ms
   [junit4]   2> 959547 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:49181/hdfs__localhost_49181__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.3_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001_tempDir-002_jetty1, hostContext=/k_t/yn, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/cores, solrconfig=solrconfig.xml, hostPort=44540}
   [junit4]   2> 959548 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@1fe91485
   [junit4]   2> 959548 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/'
   [junit4]   2> 959571 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 959573 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 959575 INFO  (zkCallback-1328-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@22fd7f85 name:ZooKeeperConnection Watcher:127.0.0.1:56104/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 959575 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 959576 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 959580 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 959580 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/solr.xml
   [junit4]   2> 959596 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/cores
   [junit4]   2> 959596 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.CoreContainer New CoreContainer 700299965
   [junit4]   2> 959596 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/]
   [junit4]   2> 959597 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.CoreContainer loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/lib
   [junit4]   2> 959597 WARN  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/lib).
   [junit4]   2> 959606 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 959607 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 959608 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 959608 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 959608 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56104/solr
   [junit4]   2> 959608 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 959608 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 959609 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 959611 INFO  (zkCallback-1330-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4fc55ad4 name:ZooKeeperConnection Watcher:127.0.0.1:56104 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 959612 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 959612 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 959614 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 959616 INFO  (zkCallback-1331-thread-1-processing-n:127.0.0.1:44540_k_t%2Fyn) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6844eedd name:ZooKeeperConnection Watcher:127.0.0.1:56104/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 959616 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 959622 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 960628 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44540_k_t%2Fyn
   [junit4]   2> 960629 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:44540_k_t%2Fyn
   [junit4]   2> 960634 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 960634 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94355766969761801-127.0.0.1:44540_k_t%2Fyn-n_0000000001
   [junit4]   2> 960635 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94355766969761796-127.0.0.1:52030_k_t%2Fyn-n_0000000000 to know if I could be the leader
   [junit4]   2> 960646 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 960646 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 960647 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.CoresLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/cores
   [junit4]   2> 960648 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.SolrCore Created CoreDescriptor: {schema=schema.xml, shard=, config=solrconfig.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, transient=false, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/cores/collection1, name=collection1, dataDir=data/, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/cores/collection1/}
   [junit4]   2> 960649 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.CoresLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1FB1272CD8B4CB56-001/shard-1-001/cores/collection1/
   [junit4]   2> 960649 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 960650 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/J0
   [junit4]   2> 960650 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1FB1272CD8B4CB56]) [n:127.0.0.1:44540_k_t%2Fyn    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 960651 INFO  (coreLoadExecutor-1317-thread-1-processing-n:127.0.0.1:44540_k_t%2Fyn) [n:127.0.0.1:44540_k_t%2Fyn c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 960651 INFO  (coreLoadExecutor-1317-thread-1-processing-n:127.0.0.1:44540_k_t%2Fyn) [n:127

[...truncated too long message...]

, TestSearchPerf, TestReplicationHandlerBackup, SpatialFilterTest, CursorMarkTest, FieldMutatingUpdateProcessorTest, DistributedSuggestComponentTest, TestElisionMultitermQuery, TestFastLRUCache, OutOfBoxZkACLAndCredentialsProvidersTest, SolrCmdDistributorTest, SimpleMLTQParserTest, TestComponentsName, TestManagedSchemaDynamicFieldResource, TestNonDefinedSimilarityFactory, TestCloudManagedSchema, CacheHeaderTest, PreAnalyzedFieldTest, TestLMJelinekMercerSimilarityFactory, TestImplicitCoreProperties, TestLMDirichletSimilarityFactory, TestSmileRequest, TestPHPSerializedResponseWriter, HdfsNNFailoverTest, CurrencyFieldOpenExchangeTest, TestLRUCache, TestHashQParserPlugin, StatelessScriptUpdateProcessorFactoryTest, TestFiltering, TestSolrCloudWithKerberosAlt, DistributedVersionInfoTest, DistribDocExpirationUpdateProcessorTest, DistributedQueueTest, ReturnFieldsTest, TestReRankQParserPlugin, DisMaxRequestHandlerTest, DistributedMLTComponentTest, TestFoldingMultitermQuery, DocumentAnalysisRequestHandlerTest, TestInitQParser, DistanceFunctionTest, TestBlobHandler, TestDownShardTolerantSearch, CollectionsAPIDistributedZkTest, BadComponentTest, TestDynamicLoading]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> 40506063 WARN  (SyncThread:0) [n:127.0.0.1:51630_ c:awholynewcollection_3 s:shard4 r:core_node12 x:awholynewcollection_3_shard4_replica1] o.a.z.s.p.FileTxnLog fsync-ing the write ahead log in SyncThread:0 took 1127ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDynamicLoading -Dtests.seed=1FB1272CD8B4CB56 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=en_NZ -Dtests.timezone=Europe/Belfast -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4]   2> 40512624 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51472) [n:127.0.0.1:51630_ c:awholynewcollection_3 s:shard4 r:core_node12 x:awholynewcollection_3_shard4_replica1] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x14f381d893c017c, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 40515461 WARN  (SyncThread:0) [n:127.0.0.1:51630_ c:awholynewcollection_3 s:shard4 r:core_node12 x:awholynewcollection_3_shard4_replica1] o.a.z.s.p.FileTxnLog fsync-ing the write ahead log in SyncThread:0 took 1137ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 40515461 ERROR (zkCallback-971-thread-29-processing-n:127.0.0.1:33250_) [n:127.0.0.1:33250_ c:awholynewstresscollection_collection1_6 s:shard4 r:core_node17 x:awholynewstresscollection_collection1_6_shard4_replica2] o.a.s.c.ZkController There was a problem finding the leader in zk:org.apache.solr.common.SolrException: CoreContainer is close
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1042)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1009)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1600)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:466)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.access$100(ZkController.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:296)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:56)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 40518301 ERROR (zkCallback-1020-thread-21-processing-n:127.0.0.1:51630_) [n:127.0.0.1:51630_ c:awholynewstresscollection_collection4_6 s:shard9 r:core_node14 x:awholynewstresscollection_collection4_6_shard9_replica2] o.a.s.c.ZkController There was a problem finding the leader in zk:org.apache.solr.common.SolrException: Could not get leader props
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1045)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1009)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1600)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:466)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.access$100(ZkController.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:296)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:56)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewstresscollection_collection3_6/leaders/shard7
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1023)
   [junit4]   2> 	... 15 more
   [junit4]   2> 
   [junit4] ERROR   0.00s J1 | TestDynamicLoading (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1FB1272CD8B4CB56]:0)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [517/531] on J1 in 332.57s, 0 tests, 1 failure <<< FAILURES!

[...truncated 2422 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build/solr-core/test/temp/junit4-J1-20150816_153625_841.sysout
   [junit4] >>> JVM J1: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/heapdumps/java_pid29070.hprof ...
   [junit4] Heap dump file created [660077402 bytes in 10.742 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 13 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:733: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/build.xml:230: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/solr/common-build.xml:524: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1449: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.3/lucene/common-build.xml:1003: There were test failures: 531 suites, 2098 tests, 4 suite-level errors, 1 error, 57 ignored (13 assumptions)

Total time: 986 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure