You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/10/10 22:45:07 UTC

[JENKINS-EA] Lucene-Solr-6.x-Linux (64bit/jdk-9-ea+138) - Build # 1919 - Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1919/
Java: 64bit/jdk-9-ea+138 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.lucene.index.TestBagOfPostings.test

Error Message:
Captured an uncaught exception in thread: Thread[id=302, name=Thread-226, state=RUNNABLE, group=TGRP-TestBagOfPostings]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=302, name=Thread-226, state=RUNNABLE, group=TGRP-TestBagOfPostings]
	at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9:24A0B821DFBDB951]:0)
Caused by: java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9]:0)
	at org.apache.lucene.index.TieredMergePolicy.findMerges(TieredMergePolicy.java:409)
	at org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2087)
	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2051)
	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4953)
	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4991)
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4982)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
	at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:111)




Build Log:
[...truncated 504 lines...]
   [junit4] Suite: org.apache.lucene.index.TestBagOfPostings
   [junit4]   2> oct. 10, 2016 5:40:55 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> AVERTISSEMENT: Uncaught exception in thread: Thread[Thread-226,5,TGRP-TestBagOfPostings]
   [junit4]   2> java.lang.AssertionError
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9]:0)
   [junit4]   2> 	at org.apache.lucene.index.TieredMergePolicy.findMerges(TieredMergePolicy.java:409)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2087)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2051)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4953)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4991)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4982)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
   [junit4]   2> 	at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:111)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBagOfPostings -Dtests.method=test -Dtests.seed=ACF487FB7141D4A9 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr-BI -Dtests.timezone=America/Porto_Velho -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   5.97s J1 | TestBagOfPostings.test <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=302, name=Thread-226, state=RUNNABLE, group=TGRP-TestBagOfPostings]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9:24A0B821DFBDB951]:0)
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9]:0)
   [junit4]    > 	at org.apache.lucene.index.TieredMergePolicy.findMerges(TieredMergePolicy.java:409)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2087)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2051)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4953)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4991)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4982)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
   [junit4]    > 	at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:111)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {field=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, maxPointsInLeafNode=153, maxMBSortInHeap=6.690350232411388, sim=ClassicSimilarity, locale=fr-BI, timezone=America/Porto_Velho
   [junit4]   2> NOTE: Linux 4.4.0-36-generic amd64/Oracle Corporation 9-ea (64-bit)/cpus=12,threads=1,free=204277432,total=508887040
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldReuse, TestLucene60PointsFormat, TestDuelingCodecs, TestIsCurrent, TestIndexWriterCommit, TestQueryRescorer, TestLSBRadixSorter, TestSimilarity2, TestPerFieldPostingsFormat, TestPostingsOffsets, TestDocumentsWriterDeleteQueue, TestDateTools, TestDemo, TestReadOnlyIndex, TestSearchForDuplicates, TestConjunctions, TestSimpleFSDirectory, MultiCollectorTest, TestBytesRefHash, TestPagedBytes, TestMixedDocValuesUpdates, TestLucene50StoredFieldsFormat, TestTotalHitCountCollector, TestAllFilesHaveChecksumFooter, TestRegexpRandom2, TestScoreCachingWrappingScorer, TestIndexWriterUnicode, TestPriorityQueue, TestBagOfPostings]
   [junit4] Completed [91/439 (1!)] on J1 in 5.98s, 1 test, 1 error <<< FAILURES!

[...truncated 62112 lines...]


Re: [JENKINS-EA] Lucene-Solr-6.x-Linux (64bit/jdk-9-ea+138) - Build # 1919 - Unstable!

Posted by Michael McCandless <lu...@mikemccandless.com>.
This is a nasty looking failure ... it tripped on an assert that I
don't think is possible :)

[junit4]   2> NOTE: reproduce with: ant test
-Dtestcase=TestBagOfPostings -Dtests.method=test
-Dtests.seed=ACF487FB7141D4A9 -Dtests.multiplier=3 -Dtests.slow=true
-Dtests.locale=fr-BI -Dtests.timezone=America/Porto_Velho
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   5.97s J1 | TestBagOfPostings.test <<<
   [junit4]    > Throwable #1:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
uncaught exception in thread: Thread[id=302, name=Thread-226,
state=RUNNABLE, group=TGRP-TestBagOfPostings]
   [junit4]    > at
__randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9:24A0B821DFBDB951]:0)
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    > at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9]:0)
   [junit4]    > at
org.apache.lucene.index.TieredMergePolicy.findMerges(TieredMergePolicy.java:409)
   [junit4]    > at
org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2087)
   [junit4]    > at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2051)
   [junit4]    > at
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4953)
   [junit4]    > at
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    > at
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4991)
   [junit4]    > at
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4982)
   [junit4]    > at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]    > at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]    > at
org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
   [junit4]    > at
org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:111)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62):
{field=PostingsFormat(name=LuceneVarGapDocFreqInterval)},
docValues:{}, maxPointsInLeafNode=153,
maxMBSortInHeap=6.690350232411388, sim=ClassicSimilarity,
locale=fr-BI, timezone=America/Porto_Velho
   [junit4]   2> NOTE: Linux 4.4.0-36-generic amd64/Oracle Corporation
9-ea (64-bit)/cpus=12,threads=1,free=204277432,total=508887040
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldReuse,
TestLucene60PointsFormat, TestDuelingCodecs, TestIsCurrent,
TestIndexWriterCommit, TestQueryRescorer, TestLSBRadixSorter,
TestSimilarity2, TestPerFieldPostingsFormat, TestPostingsOffsets,
TestDocumentsWriterDeleteQueue, TestDateTools, TestDemo,
TestReadOnlyIndex, TestSearchForDuplicates, TestConjunctions,
TestSimpleFSDirectory, MultiCollectorTest, TestBytesRefHash,
TestPagedBytes, TestMixedDocValuesUpdates,
TestLucene50StoredFieldsFormat, TestTotalHitCountCollector,
TestAllFilesHaveChecksumFooter, TestRegexpRandom2,
TestScoreCachingWrappingScorer, TestIndexWriterUnicode,
TestPriorityQueue, TestBagOfPostings]

It's this assert:

          // We should never see an empty candidate: we iterated over
maxMergeAtOnce
          // segments, and already pre-excluded the too-large segments:
          assert candidate.size() > 0;

candidate is an ArrayList, and it should always have at least one
element based on the (admittedly rather hairy) logic above ... I
suspect there is an exciting Java 9 hotspot bug here.  The failure
doesn't repro on Java 1.8.0_101 nor Java 9-ea+139.

Mike McCandless

http://blog.mikemccandless.com


On Mon, Oct 10, 2016 at 6:45 PM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1919/
> Java: 64bit/jdk-9-ea+138 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
>
> 1 tests failed.
> FAILED:  org.apache.lucene.index.TestBagOfPostings.test
>
> Error Message:
> Captured an uncaught exception in thread: Thread[id=302, name=Thread-226, state=RUNNABLE, group=TGRP-TestBagOfPostings]
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=302, name=Thread-226, state=RUNNABLE, group=TGRP-TestBagOfPostings]
>         at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9:24A0B821DFBDB951]:0)
> Caused by: java.lang.AssertionError
>         at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9]:0)
>         at org.apache.lucene.index.TieredMergePolicy.findMerges(TieredMergePolicy.java:409)
>         at org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2087)
>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2051)
>         at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4953)
>         at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4991)
>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4982)
>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
>         at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
>         at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:111)
>
>
>
>
> Build Log:
> [...truncated 504 lines...]
>    [junit4] Suite: org.apache.lucene.index.TestBagOfPostings
>    [junit4]   2> oct. 10, 2016 5:40:55 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
>    [junit4]   2> AVERTISSEMENT: Uncaught exception in thread: Thread[Thread-226,5,TGRP-TestBagOfPostings]
>    [junit4]   2> java.lang.AssertionError
>    [junit4]   2>        at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9]:0)
>    [junit4]   2>        at org.apache.lucene.index.TieredMergePolicy.findMerges(TieredMergePolicy.java:409)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2087)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2051)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4953)
>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4991)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4982)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
>    [junit4]   2>        at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
>    [junit4]   2>        at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:111)
>    [junit4]   2>
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBagOfPostings -Dtests.method=test -Dtests.seed=ACF487FB7141D4A9 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr-BI -Dtests.timezone=America/Porto_Velho -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>    [junit4] ERROR   5.97s J1 | TestBagOfPostings.test <<<
>    [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=302, name=Thread-226, state=RUNNABLE, group=TGRP-TestBagOfPostings]
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9:24A0B821DFBDB951]:0)
>    [junit4]    > Caused by: java.lang.AssertionError
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([ACF487FB7141D4A9]:0)
>    [junit4]    >        at org.apache.lucene.index.TieredMergePolicy.findMerges(TieredMergePolicy.java:409)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2087)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2051)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4953)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4991)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4982)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
>    [junit4]    >        at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
>    [junit4]    >        at org.apache.lucene.index.TestBagOfPostings$1.run(TestBagOfPostings.java:111)
>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {field=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, maxPointsInLeafNode=153, maxMBSortInHeap=6.690350232411388, sim=ClassicSimilarity, locale=fr-BI, timezone=America/Porto_Velho
>    [junit4]   2> NOTE: Linux 4.4.0-36-generic amd64/Oracle Corporation 9-ea (64-bit)/cpus=12,threads=1,free=204277432,total=508887040
>    [junit4]   2> NOTE: All tests run in this JVM: [TestFieldReuse, TestLucene60PointsFormat, TestDuelingCodecs, TestIsCurrent, TestIndexWriterCommit, TestQueryRescorer, TestLSBRadixSorter, TestSimilarity2, TestPerFieldPostingsFormat, TestPostingsOffsets, TestDocumentsWriterDeleteQueue, TestDateTools, TestDemo, TestReadOnlyIndex, TestSearchForDuplicates, TestConjunctions, TestSimpleFSDirectory, MultiCollectorTest, TestBytesRefHash, TestPagedBytes, TestMixedDocValuesUpdates, TestLucene50StoredFieldsFormat, TestTotalHitCountCollector, TestAllFilesHaveChecksumFooter, TestRegexpRandom2, TestScoreCachingWrappingScorer, TestIndexWriterUnicode, TestPriorityQueue, TestBagOfPostings]
>    [junit4] Completed [91/439 (1!)] on J1 in 5.98s, 1 test, 1 error <<< FAILURES!
>
> [...truncated 62112 lines...]
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-6.x-Linux (64bit/jdk1.8.0_102) - Build # 1920 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1920/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:40653

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:40653
	at __randomizedtesting.SeedInfo.seed([AF74C4E383E2F15A:2720FB392D1E9CA2]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:604)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1292)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1062)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1004)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1553)
	at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForHashRouter(CustomCollectionTest.java:332)
	at org.apache.solr.cloud.CustomCollectionTest.test(CustomCollectionTest.java:98)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:498)
	... 53 more




Build Log:
[...truncated 12154 lines...]
   [junit4] Suite: org.apache.solr.cloud.CustomCollectionTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/init-core-data-001
   [junit4]   2> 1293314 INFO  (SUITE-CustomCollectionTest-seed#[AF74C4E383E2F15A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 1293314 INFO  (SUITE-CustomCollectionTest-seed#[AF74C4E383E2F15A]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1293316 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1293316 INFO  (Thread-1603) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1293316 INFO  (Thread-1603) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1293416 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.ZkTestServer start zk server on port:37043
   [junit4]   2> 1293422 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1293423 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1293424 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1293424 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1293424 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1293425 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1293425 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1293426 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1293426 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1293427 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1293427 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1293484 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/control-001/cores/collection1
   [junit4]   2> 1293485 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1293486 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@a229891{/,null,AVAILABLE}
   [junit4]   2> 1293487 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.ServerConnector Started ServerConnector@278dfd45{HTTP/1.1,[http/1.1]}{127.0.0.1:36692}
   [junit4]   2> 1293487 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server Started @1295645ms
   [junit4]   2> 1293487 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/tempDir-001/control/data, hostContext=/, hostPort=36692, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/control-001/cores}
   [junit4]   2> 1293487 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.3.0
   [junit4]   2> 1293488 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1293488 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1293488 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-10-11T01:22:13.219Z
   [junit4]   2> 1293489 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1293490 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/control-001/solr.xml
   [junit4]   2> 1293494 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1293495 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37043/solr
   [junit4]   2> 1293503 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:36692_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36692_
   [junit4]   2> 1293504 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:36692_    ] o.a.s.c.Overseer Overseer (id=96740896476233732-127.0.0.1:36692_-n_0000000000) starting
   [junit4]   2> 1293508 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:36692_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36692_
   [junit4]   2> 1293546 INFO  (zkCallback-1232-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1293554 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:36692_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/control-001/cores
   [junit4]   2> 1293554 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:36692_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1293555 INFO  (OverseerStateUpdate-96740896476233732-127.0.0.1:36692_-n_0000000000) [n:127.0.0.1:36692_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1294564 WARN  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1294566 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0
   [junit4]   2> 1294582 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1294653 WARN  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1294654 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1294662 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1294662 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/control-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/control-001/cores/collection1/data/]
   [junit4]   2> 1294662 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77ca07c6
   [junit4]   2> 1294663 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=52.0380859375, floorSegmentMB=0.919921875, forceMergeDeletesPctAllowed=27.48832057185957, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8980320069340372
   [junit4]   2> 1294666 WARN  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1294673 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1294673 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1294675 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1294675 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1294675 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6081733196250501]
   [junit4]   2> 1294676 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@288c016b[collection1] main]
   [junit4]   2> 1294676 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1294676 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1294676 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1294677 INFO  (searcherExecutor-3505-thread-1-processing-n:127.0.0.1:36692_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@288c016b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1294677 INFO  (coreLoadExecutor-3504-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547854345045803008
   [junit4]   2> 1294682 INFO  (coreZkRegister-3497-thread-1-processing-n:127.0.0.1:36692_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1294682 INFO  (coreZkRegister-3497-thread-1-processing-n:127.0.0.1:36692_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1294682 INFO  (coreZkRegister-3497-thread-1-processing-n:127.0.0.1:36692_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36692/collection1/
   [junit4]   2> 1294682 INFO  (coreZkRegister-3497-thread-1-processing-n:127.0.0.1:36692_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1294682 INFO  (coreZkRegister-3497-thread-1-processing-n:127.0.0.1:36692_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:36692/collection1/ has no replicas
   [junit4]   2> 1294684 INFO  (coreZkRegister-3497-thread-1-processing-n:127.0.0.1:36692_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:36692/collection1/ shard1
   [junit4]   2> 1294834 INFO  (coreZkRegister-3497-thread-1-processing-n:127.0.0.1:36692_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:36692_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1295057 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1295058 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1295114 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-1-001/cores/collection1
   [junit4]   2> 1295115 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-1-001
   [junit4]   2> 1295115 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1295116 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@26dfb2cb{/,null,AVAILABLE}
   [junit4]   2> 1295116 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4d5f08ab{HTTP/1.1,[http/1.1]}{127.0.0.1:40653}
   [junit4]   2> 1295117 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server Started @1297274ms
   [junit4]   2> 1295117 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=40653, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-1-001/cores}
   [junit4]   2> 1295117 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.3.0
   [junit4]   2> 1295117 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1295117 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1295117 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-10-11T01:22:14.848Z
   [junit4]   2> 1295119 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1295119 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-1-001/solr.xml
   [junit4]   2> 1295124 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1295124 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37043/solr
   [junit4]   2> 1295129 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:40653_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1295131 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:40653_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40653_
   [junit4]   2> 1295132 INFO  (zkCallback-1241-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1295132 INFO  (zkCallback-1232-thread-2-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1295132 INFO  (zkCallback-1236-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1295139 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:40653_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-1-001/cores
   [junit4]   2> 1295139 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:40653_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1295140 INFO  (OverseerStateUpdate-96740896476233732-127.0.0.1:36692_-n_0000000000) [n:127.0.0.1:36692_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1296185 WARN  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1296187 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0
   [junit4]   2> 1296204 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1296289 WARN  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1296291 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1296297 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1296297 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-1-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 1296297 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77ca07c6
   [junit4]   2> 1296298 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=52.0380859375, floorSegmentMB=0.919921875, forceMergeDeletesPctAllowed=27.48832057185957, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8980320069340372
   [junit4]   2> 1296301 WARN  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1296307 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1296307 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1296309 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1296309 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1296310 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6081733196250501]
   [junit4]   2> 1296310 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@179cf3ba[collection1] main]
   [junit4]   2> 1296310 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1296311 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1296311 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1296311 INFO  (searcherExecutor-3516-thread-1-processing-n:127.0.0.1:40653_ x:collection1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@179cf3ba[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1296312 INFO  (coreLoadExecutor-3515-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547854346760224768
   [junit4]   2> 1296316 INFO  (coreZkRegister-3510-thread-1-processing-n:127.0.0.1:40653_ x:collection1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1296316 INFO  (coreZkRegister-3510-thread-1-processing-n:127.0.0.1:40653_ x:collection1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1296316 INFO  (coreZkRegister-3510-thread-1-processing-n:127.0.0.1:40653_ x:collection1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40653/collection1/
   [junit4]   2> 1296316 INFO  (coreZkRegister-3510-thread-1-processing-n:127.0.0.1:40653_ x:collection1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1296316 INFO  (coreZkRegister-3510-thread-1-processing-n:127.0.0.1:40653_ x:collection1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40653/collection1/ has no replicas
   [junit4]   2> 1296317 INFO  (coreZkRegister-3510-thread-1-processing-n:127.0.0.1:40653_ x:collection1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40653/collection1/ shard2
   [junit4]   2> 1296468 INFO  (coreZkRegister-3510-thread-1-processing-n:127.0.0.1:40653_ x:collection1 s:shard2 c:collection1 r:core_node1) [n:127.0.0.1:40653_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1296695 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-2-001/cores/collection1
   [junit4]   2> 1296695 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-2-001
   [junit4]   2> 1296696 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1296697 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1ac98b7a{/,null,AVAILABLE}
   [junit4]   2> 1296697 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.ServerConnector Started ServerConnector@64bf820d{HTTP/1.1,[http/1.1]}{127.0.0.1:44895}
   [junit4]   2> 1296698 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server Started @1298856ms
   [junit4]   2> 1296698 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=44895, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-2-001/cores}
   [junit4]   2> 1296698 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.3.0
   [junit4]   2> 1296698 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1296698 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1296698 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-10-11T01:22:16.429Z
   [junit4]   2> 1296700 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1296700 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-2-001/solr.xml
   [junit4]   2> 1296705 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1296706 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37043/solr
   [junit4]   2> 1296711 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:44895_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1296713 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:44895_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44895_
   [junit4]   2> 1296713 INFO  (zkCallback-1236-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1296713 INFO  (zkCallback-1232-thread-2-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1296714 INFO  (zkCallback-1241-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1296719 INFO  (zkCallback-1247-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1296725 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:44895_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-2-001/cores
   [junit4]   2> 1296726 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:44895_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1296727 INFO  (OverseerStateUpdate-96740896476233732-127.0.0.1:36692_-n_0000000000) [n:127.0.0.1:36692_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1297744 WARN  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1297745 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0
   [junit4]   2> 1297756 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1297824 WARN  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1297826 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1297832 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1297832 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-2-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-2-001/cores/collection1/data/]
   [junit4]   2> 1297832 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77ca07c6
   [junit4]   2> 1297833 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=52.0380859375, floorSegmentMB=0.919921875, forceMergeDeletesPctAllowed=27.48832057185957, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8980320069340372
   [junit4]   2> 1297836 WARN  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1297843 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1297843 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1297845 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1297845 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1297846 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6081733196250501]
   [junit4]   2> 1297846 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7e01e94a[collection1] main]
   [junit4]   2> 1297847 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1297847 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1297847 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1297848 INFO  (searcherExecutor-3527-thread-1-processing-n:127.0.0.1:44895_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7e01e94a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1297848 INFO  (coreLoadExecutor-3526-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547854348370837504
   [junit4]   2> 1297853 INFO  (coreZkRegister-3521-thread-1-processing-n:127.0.0.1:44895_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1297853 INFO  (coreZkRegister-3521-thread-1-processing-n:127.0.0.1:44895_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1297853 INFO  (coreZkRegister-3521-thread-1-processing-n:127.0.0.1:44895_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44895/collection1/
   [junit4]   2> 1297853 INFO  (coreZkRegister-3521-thread-1-processing-n:127.0.0.1:44895_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1297853 INFO  (coreZkRegister-3521-thread-1-processing-n:127.0.0.1:44895_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:44895/collection1/ has no replicas
   [junit4]   2> 1297855 INFO  (coreZkRegister-3521-thread-1-processing-n:127.0.0.1:44895_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44895/collection1/ shard1
   [junit4]   2> 1298005 INFO  (coreZkRegister-3521-thread-1-processing-n:127.0.0.1:44895_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:44895_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1298284 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-3-001/cores/collection1
   [junit4]   2> 1298285 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-3-001
   [junit4]   2> 1298286 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1298287 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@72e06f47{/,null,AVAILABLE}
   [junit4]   2> 1298287 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.ServerConnector Started ServerConnector@59294f98{HTTP/1.1,[http/1.1]}{127.0.0.1:34243}
   [junit4]   2> 1298288 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server Started @1300445ms
   [junit4]   2> 1298288 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=34243, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-3-001/cores}
   [junit4]   2> 1298288 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.3.0
   [junit4]   2> 1298288 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1298288 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1298288 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-10-11T01:22:18.019Z
   [junit4]   2> 1298291 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1298291 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-3-001/solr.xml
   [junit4]   2> 1298299 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1298299 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37043/solr
   [junit4]   2> 1298304 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:34243_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1298307 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:34243_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34243_
   [junit4]   2> 1298307 INFO  (zkCallback-1232-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1298308 INFO  (zkCallback-1253-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1298308 INFO  (zkCallback-1241-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1298308 INFO  (zkCallback-1247-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1298308 INFO  (zkCallback-1236-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1298322 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:34243_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-3-001/cores
   [junit4]   2> 1298322 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:34243_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1298323 INFO  (OverseerStateUpdate-96740896476233732-127.0.0.1:36692_-n_0000000000) [n:127.0.0.1:36692_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1299330 WARN  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1299331 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0
   [junit4]   2> 1299342 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1299408 WARN  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1299410 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1299416 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1299416 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-3-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-3-001/cores/collection1/data/]
   [junit4]   2> 1299416 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77ca07c6
   [junit4]   2> 1299417 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=52.0380859375, floorSegmentMB=0.919921875, forceMergeDeletesPctAllowed=27.48832057185957, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8980320069340372
   [junit4]   2> 1299420 WARN  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1299428 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1299428 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1299429 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1299430 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1299430 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6081733196250501]
   [junit4]   2> 1299431 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@366940a6[collection1] main]
   [junit4]   2> 1299431 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1299431 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1299431 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1299432 INFO  (searcherExecutor-3538-thread-1-processing-n:127.0.0.1:34243_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@366940a6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1299432 INFO  (coreLoadExecutor-3537-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547854350031781888
   [junit4]   2> 1299436 INFO  (coreZkRegister-3532-thread-1-processing-n:127.0.0.1:34243_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 1299437 INFO  (updateExecutor-1250-thread-1-processing-n:127.0.0.1:34243_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 1299437 INFO  (recoveryExecutor-1251-thread-1-processing-n:127.0.0.1:34243_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 1299438 INFO  (recoveryExecutor-1251-thread-1-processing-n:127.0.0.1:34243_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 1299438 INFO  (recoveryExecutor-1251-thread-1-processing-n:127.0.0.1:34243_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 1299438 INFO  (recoveryExecutor-1251-thread-1-processing-n:127.0.0.1:34243_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 1299438 INFO  (recoveryExecutor-1251-thread-1-processing-n:127.0.0.1:34243_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:40653/collection1/] and I am [http://127.0.0.1:34243/collection1/]
   [junit4]   2> 1299439 INFO  (recoveryExecutor-1251-thread-1-processing-n:127.0.0.1:34243_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:34243_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:40653]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:34243_&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 1299440 INFO  (qtp1586429739-7290) [n:127.0.0.1:40653_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 1299441 INFO  (qtp1586429739-7290) [n:127.0.0.1:40653_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard2 of collection1) have state: recovering
   [junit4]   2> 1299441 INFO  (qtp1586429739-7290) [n:127.0.0.1:40653_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:34243_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:34243","node_name":"127.0.0.1:34243_","state":"down"}
   [junit4]   2> 1299884 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-4-001/cores/collection1
   [junit4]   2> 1299885 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-4-001
   [junit4]   2> 1299886 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1299886 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6f2443e5{/,null,AVAILABLE}
   [junit4]   2> 1299887 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.ServerConnector Started ServerConnector@22d80423{HTTP/1.1,[http/1.1]}{127.0.0.1:45816}
   [junit4]   2> 1299888 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.e.j.s.Server Started @1302045ms
   [junit4]   2> 1299888 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/tempDir-001/jetty4, solrconfig=solrconfig.xml, hostContext=/, hostPort=45816, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-4-001/cores}
   [junit4]   2> 1299888 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.3.0
   [junit4]   2> 1299888 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1299888 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1299888 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-10-11T01:22:19.619Z
   [junit4]   2> 1299890 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1299890 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-4-001/solr.xml
   [junit4]   2> 1299916 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1299916 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37043/solr
   [junit4]   2> 1299921 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:45816_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1299922 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:45816_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45816_
   [junit4]   2> 1299923 INFO  (zkCallback-1241-thread-1-processing-n:127.0.0.1:40653_) [n:127.0.0.1:40653_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1299923 INFO  (zkCallback-1253-thread-1-processing-n:127.0.0.1:34243_) [n:127.0.0.1:34243_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1299923 INFO  (zkCallback-1247-thread-1-processing-n:127.0.0.1:44895_) [n:127.0.0.1:44895_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1299923 INFO  (zkCallback-1236-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1299923 INFO  (zkCallback-1232-thread-3-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1299924 INFO  (zkCallback-1260-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1299932 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:45816_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-4-001/cores
   [junit4]   2> 1299932 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [n:127.0.0.1:45816_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1299932 INFO  (OverseerStateUpdate-96740896476233732-127.0.0.1:36692_-n_0000000000) [n:127.0.0.1:36692_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1300441 INFO  (qtp1586429739-7290) [n:127.0.0.1:40653_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:34243_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:34243","node_name":"127.0.0.1:34243_","state":"recovering"}
   [junit4]   2> 1300441 INFO  (qtp1586429739-7290) [n:127.0.0.1:40653_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 1300441 INFO  (qtp1586429739-7290) [n:127.0.0.1:40653_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:34243_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
   [junit4]   2> 1300940 WARN  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1300940 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.3.0
   [junit4]   2> 1300952 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1301021 WARN  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1301023 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1301029 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1301029 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-4-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001/shard-4-001/cores/collection1/data/]
   [junit4]   2> 1301029 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77ca07c6
   [junit4]   2> 1301030 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=52.0380859375, floorSegmentMB=0.919921875, forceMergeDeletesPctAllowed=27.48832057185957, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8980320069340372
   [junit4]   2> 1301033 WARN  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1301039 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1301039 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1301041 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1301041 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1301041 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6081733196250501]
   [junit4]   2> 1301042 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@ef29978[collection1] main]
   [junit4]   2> 1301042 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1301042 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1301043 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1301043 INFO  (searcherExecutor-3549-thread-1-processing-n:127.0.0.1:45816_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@ef29978[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1301043 INFO  (coreLoadExecutor-3548-thread-1-processing-n:127.0.0.1:45816_) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547854351721037824
   [junit4]   2> 1301046 INFO  (coreZkRegister-3543-thread-1-processing-n:127.0.0.1:45816_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 1301047 INFO  (updateExecutor-1257-thread-1-processing-n:127.0.0.1:45816_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 1301047 INFO  (recoveryExecutor-1258-thread-1-processing-n:127.0.0.1:45816_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 1301047 INFO  (recoveryExecutor-1258-thread-1-processing-n:127.0.0.1:45816_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 1301047 INFO  (recoveryExecutor-1258-thread-1-processing-n:127.0.0.1:45816_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 1301047 INFO  (recoveryExecutor-1258-thread-1-processing-n:127.0.0.1:45816_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 1301047 INFO  (recoveryExecutor-1258-thread-1-processing-n:127.0.0.1:45816_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:44895/collection1/] and I am [http://127.0.0.1:45816/collection1/]
   [junit4]   2> 1301049 INFO  (recoveryExecutor-1258-thread-1-processing-n:127.0.0.1:45816_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:45816_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:44895]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:45816_&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 1301050 INFO  (qtp137998798-7318) [n:127.0.0.1:44895_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 1301051 INFO  (qtp137998798-7318) [n:127.0.0.1:44895_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 1301051 INFO  (qtp137998798-7318) [n:127.0.0.1:44895_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:45816_, coreNodeName=core_node4, onlyIfActiveCheckResult=false, nodeProps: core_node4:{"core":"collection1","base_url":"http://127.0.0.1:45816","node_name":"127.0.0.1:45816_","state":"down"}
   [junit4]   2> 1301434 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 1301436 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 1301437 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Created commonCloudSolrClient with updatesToLeaders=true and parallelUpdates=false
   [junit4]   2> 1301438 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection with stateFormat=1: implicitcoll0
   [junit4]   2> 1301439 INFO  (TEST-CustomCollectionTest.test-seed#[AF74C4E383E2F15A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
   [junit4]   2> 1301440 INFO  (qtp1586429739-7290) [n:127.0.0.1:40653_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=5&shards=a,b,c&maxShardsPerNode=5&router.name=implicit&name=implicitcoll0&action=CREATE&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1301441 INFO  (OverseerThreadFactory-3502-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_    ] o.a.s.c.CreateCollectionCmd Create collection implicitcoll0
   [junit4]   2> 1301442 INFO  (OverseerThreadFactory-3502-thread-1-processing-n:127.0.0.1:36692_) [n:127.0.0.1:36692_    ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf1
   [junit4]   2> 1301649 INFO  (qtp1379025071-7256) [n:127.0.0.1:36692_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_c_replica5&action=CREATE&numShards=3&collection=implicitcoll0&shard=c&wt=javabin&version=2
   [junit4]   2> 1301649 INFO  (qtp1586429739-7292) [n:127.0.0.1:40653_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_a_replica3&action=CREATE&numShards=3&collection=implicitcoll0&shard=a&wt=javabin&version=2
   [junit4]   2> 1301650 INFO  (qtp1379025071-7252) [n:127.0.0.1:36692_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_b_replica5&action=CREATE&numShards=3&collection=implicitcoll0&shard=b&wt=javabin&version=2
   [junit4]   2> 1301649 INFO  (qtp137998798-7320) [n:127.0.0.1:44895_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_c_replica2&action=CREATE&numShards=3&collection=implicitcoll0&shard=c&wt=javabin&version=2
   [junit4]   2> 1301650 INFO  (qtp1379025071-7255) [n:127.0.0.1:36692_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_a_replica5&action=CREATE&numShards=3&collection=implicitcoll0&shard=a&wt=javabin&version=2
   [junit4]   2> 1301678 INFO  (qtp137998798-7314) [n:127.0.0.1:44895_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_a_replica2&action=CREATE&numShards=3&collection=implicitcoll0&shard=a&wt=javabin&version=2
   [junit4]   2> 1301678 INFO  (qtp1586429739-7288) [n:127.0.0.1:40653_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_c_replica3&action=CREATE&numShards=3&collection=implicitcoll0&shard=c&wt=javabin&version=2
   [junit4]   2> 1301679 INFO  (qtp1994205723-7378) [n:127.0.0.1:45816_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_b_replica4&action=CREATE&numShards=3&collection=implicitcoll0&shard=b&wt=javabin&version=2
   [junit4]   2> 1301679 INFO  (qtp1994205723-7377) [n:127.0.0.1:45816_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_a_replica4&action=CREATE&numShards=3&collection=implicitcoll0&shard=a&wt=javabin&version=2
   [junit4]   2> 1301679 INFO  (qtp1586429739-7285) [n:127.0.0.1:40653_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_b_replica3&action=CREATE&numShards=3&collection=implicitcoll0&shard=b&wt=javabin&version=2
   [junit4]   2> 1301679 INFO  (qtp1883725684-7348) [n:127.0.0.1:34243_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_b_replica1&action=CREATE&numShards=3&collection=implicitcoll0&shard=b&wt=javabin&version=2
   [junit4]   2> 1301680 INFO  (qtp137998798-7438) [n:127.0.0.1:44895_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_b_replica2&action=CREATE&numShards=3&collection=implicitcoll0&shard=b&wt=javabin&version=2
   [junit4]   2> 1301681 INFO  (qtp1994205723-7381) [n:127.0.0.1:45816_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_c_replica4&action=CREATE&numShards=3&collection=implicitcoll0&shard=c&wt=javabin&version=2
   [junit4]   2> 1301681 INFO  (qtp1883725684-7346) [n:127.0.0.1:34243_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=implicitcoll0_a_replica1&action=CREATE&numShards=3&collection=implicitcoll0&shard=a&wt=javabin&version=2
   [junit4]   2> 1301682 INFO  (qtp1883725684-7347) [n:127.0.0.1:34243_    ] o.a.s.h.a.CoreAdminOperation core cre

[...truncated too long message...]

ollwithShardField/leader_elect/c/election/96740896476233739-core_node12-n_0000000001
   [junit4]   2> 	2	/solr/collections/implicitcoll1/leader_elect/a/election/96740896476233745-core_node4-n_0000000002
   [junit4]   2> 	2	/solr/collections/implicitcoll5/leader_elect/a/election/96740896476233739-core_node8-n_0000000002
   [junit4]   2> 	2	/solr/collections/implicitcoll0/leader_elect/b/election/96740896476233742-core_node10-n_0000000002
   [junit4]   2> 	2	/solr/collections/implicitcoll3/leader_elect/c/election/96740896476233742-core_node2-n_0000000002
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	136	/solr/overseer/queue
   [junit4]   2> 	89	/solr/overseer/collection-queue-work
   [junit4]   2> 	11	/solr/live_nodes
   [junit4]   2> 	11	/solr/collections
   [junit4]   2> 	10	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CustomCollectionTest -Dtests.method=test -Dtests.seed=AF74C4E383E2F15A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-CA -Dtests.timezone=America/Manaus -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    253s J2 | CustomCollectionTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:40653
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AF74C4E383E2F15A:2720FB392D1E9CA2]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:604)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1292)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1062)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1004)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1553)
   [junit4]    > 	at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForHashRouter(CustomCollectionTest.java:332)
   [junit4]    > 	at org.apache.solr.cloud.CustomCollectionTest.test(CustomCollectionTest.java:98)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:170)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]    > 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:498)
   [junit4]    > 	... 53 more
   [junit4]   2> 1546611 INFO  (SUITE-CustomCollectionTest-seed#[AF74C4E383E2F15A]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CustomCollectionTest_AF74C4E383E2F15A-001
   [junit4]   2> Oct 11, 2016 1:26:26 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {shard_s=PostingsFormat(name=Memory doPackFST= false), other_tl1=Lucene50(blocksize=128), range_facet_l_dv=PostingsFormat(name=MockRandom), multiDefault=PostingsFormat(name=Memory doPackFST= false), intDefault=Lucene50(blocksize=128), a_i1=PostingsFormat(name=Memory doPackFST= false), range_facet_l=PostingsFormat(name=Memory doPackFST= false), _version_=Lucene50(blocksize=128), a_t=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=MockRandom), range_facet_i_dv=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=LuceneFixedGap), _route_=PostingsFormat(name=LuceneFixedGap), timestamp=PostingsFormat(name=Memory doPackFST= false)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Asserting), range_facet_i_dv=DocValuesFormat(name=Lucene54), timestamp=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=1854, maxMBSortInHeap=6.6499957062770285, sim=ClassicSimilarity, locale=en-CA, timezone=America/Manaus
   [junit4]   2> NOTE: Linux 4.4.0-36-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=12,threads=1,free=202071440,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestRestoreCore, ResponseLogComponentTest, TestAuthenticationFramework, TestSchemaResource, TestCollationField, HdfsDirectoryFactoryTest, TestUniqueKeyFieldResource, AutoCommitTest, WrapperMergePolicyFactoryTest, BitVectorTest, TestSort, DistributedExpandComponentTest, TestSSLRandomization, CoreMergeIndexesAdminHandlerTest, ConnectionManagerTest, DirectUpdateHandlerOptimizeTest, TolerantUpdateProcessorTest, XmlUpdateRequestHandlerTest, TestSimpleQParserPlugin, TestScoreJoinQPNoScore, SolrCLIZkUtilsTest, TestFieldCacheSortRandom, UpdateParamsTest, TimeZoneUtilsTest, DistributedSuggestComponentTest, PreAnalyzedFieldTest, TestDistribIDF, PrimUtilsTest, TestRecovery, DeleteInactiveReplicaTest, TestRandomCollapseQParserPlugin, BasicDistributedZk2Test, TestTolerantSearch, TestSha256AuthenticationProvider, TestRecoveryHdfs, OverseerStatusTest, TestAnalyzeInfixSuggestions, TestMacroExpander, ReplicationFactorTest, TestLFUCache, UnloadDistributedZkTest, JavabinLoaderTest, TestSolrQueryParser, ConnectionReuseTest, RequestHandlersTest, TestFieldTypeCollectionResource, MergeStrategyTest, TestDFISimilarityFactory, TestFieldCacheReopen, TestPerFieldSimilarityWithDefaultOverride, TestSolrFieldCacheMBean, TestConfigSetsAPI, TestSolrJ, TestLeaderElectionZkExpiry, TestZkChroot, TestConfigSets, PluginInfoTest, TestCryptoKeys, TestSolrDynamicMBean, TestChildDocTransformer, StatsComponentTest, SSLMigrationTest, HdfsLockFactoryTest, TestAddFieldRealTimeGet, HttpPartitionTest, DOMUtilTest, BlockDirectoryTest, TestCoreDiscovery, TestLMJelinekMercerSimilarityFactory, TestStressReorder, ShowFileRequestHandlerTest, TestIndexingPerformance, TestExclusionRuleCollectionAccess, PingRequestHandlerTest, BinaryUpdateRequestHandlerTest, ParsingFieldUpdateProcessorsTest, CdcrBootstrapTest, TestReload, TestObjectReleaseTracker, AssignTest, TestTestInjection, TestRestManager, HdfsChaosMonkeySafeLeaderTest, SmileWriterTest, HdfsRecoverLeaseTest, CloneFieldUpdateProcessorFactoryTest, JSONWriterTest, CoreSorterTest, PrimitiveFieldTypeTest, CollectionsAPISolrJTest, TestPostingsSolrHighlighter, TestWriterPerf, OpenExchangeRatesOrgProviderTest, HardAutoCommitTest, TestLegacyFieldCache, TestRandomFlRTGCloud, FullSolrCloudDistribCmdsTest, StatelessScriptUpdateProcessorFactoryTest, RecoveryZkTest, TestSolrDeletionPolicy1, TestFilteredDocIdSet, MinimalSchemaTest, FieldMutatingUpdateProcessorTest, ConcurrentDeleteAndCreateCollectionTest, ConfigSetsAPITest, CustomCollectionTest]
   [junit4] Completed [471/637 (1!)] on J2 in 254.06s, 1 test, 1 error <<< FAILURES!

[...truncated 60896 lines...]