You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/02/22 09:02:11 UTC

[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 713 - Failure!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/713/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 75495 lines...]
changes-to-html:
    [mkdir] Created dir: /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/docs/changes
     [exec] Section 'Optimizations' appears more than once under release '6.5.0' at /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/site/changes/changes2html.pl line 136.

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:775: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:101: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:654: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:629: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:2518: exec returned: 25

Total time: 116 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 715 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/715/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI

Error Message:
Could not find collection : implicitcoll

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : implicitcoll
	at __randomizedtesting.SeedInfo.seed([D1E37C890235A44:67FFB9A3ADB9EC3C]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:194)
	at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:245)
	at org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI(CustomCollectionTest.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestSegmentSorting.testAtomicUpdateOfSegmentSortField

Error Message:
Could not find collection:testAtomicUpdateOfSegmentSortField

Stack Trace:
java.lang.AssertionError: Could not find collection:testAtomicUpdateOfSegmentSortField
	at __randomizedtesting.SeedInfo.seed([D1E37C890235A44:EC0C10CD4666C1FC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
	at org.apache.solr.cloud.TestSegmentSorting.createCollection(TestSegmentSorting.java:89)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:941)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestSegmentSorting.testSegmentTerminateEarly

Error Message:
expected:<COMPLETED> but was:<FAILED>

Stack Trace:
java.lang.AssertionError: expected:<COMPLETED> but was:<FAILED>
	at __randomizedtesting.SeedInfo.seed([D1E37C890235A44:DDB8F0654C71CE2E]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.cloud.TestSegmentSorting.createCollection(TestSegmentSorting.java:85)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:941)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12465 lines...]
   [junit4] Suite: org.apache.solr.cloud.CustomCollectionTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/init-core-data-001
   [junit4]   2> 2217796 INFO  (SUITE-CustomCollectionTest-seed#[D1E37C890235A44]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 2217801 INFO  (SUITE-CustomCollectionTest-seed#[D1E37C890235A44]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2217808 INFO  (SUITE-CustomCollectionTest-seed#[D1E37C890235A44]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001
   [junit4]   2> 2217808 INFO  (SUITE-CustomCollectionTest-seed#[D1E37C890235A44]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2217808 INFO  (Thread-4121) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2217808 INFO  (Thread-4121) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2217912 INFO  (SUITE-CustomCollectionTest-seed#[D1E37C890235A44]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:51531
   [junit4]   2> 2217943 INFO  (jetty-launcher-3075-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2217943 INFO  (jetty-launcher-3075-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2217944 INFO  (jetty-launcher-3075-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2217944 INFO  (jetty-launcher-3075-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2217946 INFO  (jetty-launcher-3075-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@c6421eb{/solr,null,AVAILABLE}
   [junit4]   2> 2217947 INFO  (jetty-launcher-3075-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@786934fb{SSL,[ssl, http/1.1]}{127.0.0.1:51533}
   [junit4]   2> 2217947 INFO  (jetty-launcher-3075-thread-4) [    ] o.e.j.s.Server Started @2223395ms
   [junit4]   2> 2217948 INFO  (jetty-launcher-3075-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51533}
   [junit4]   2> 2217948 ERROR (jetty-launcher-3075-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2217948 INFO  (jetty-launcher-3075-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 2217949 INFO  (jetty-launcher-3075-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2217949 INFO  (jetty-launcher-3075-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2217949 INFO  (jetty-launcher-3075-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-23T05:18:52.755Z
   [junit4]   2> 2217951 INFO  (jetty-launcher-3075-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3b6095f8{/solr,null,AVAILABLE}
   [junit4]   2> 2217952 INFO  (jetty-launcher-3075-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@51dab59d{SSL,[ssl, http/1.1]}{127.0.0.1:51534}
   [junit4]   2> 2217952 INFO  (jetty-launcher-3075-thread-3) [    ] o.e.j.s.Server Started @2223399ms
   [junit4]   2> 2217952 INFO  (jetty-launcher-3075-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51534}
   [junit4]   2> 2217952 ERROR (jetty-launcher-3075-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2217953 INFO  (jetty-launcher-3075-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 2217953 INFO  (jetty-launcher-3075-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2217953 INFO  (jetty-launcher-3075-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2217953 INFO  (jetty-launcher-3075-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-23T05:18:52.759Z
   [junit4]   2> 2217954 INFO  (jetty-launcher-3075-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5d365e44{/solr,null,AVAILABLE}
   [junit4]   2> 2217955 INFO  (jetty-launcher-3075-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6f5a945e{SSL,[ssl, http/1.1]}{127.0.0.1:51535}
   [junit4]   2> 2217955 INFO  (jetty-launcher-3075-thread-1) [    ] o.e.j.s.Server Started @2223402ms
   [junit4]   2> 2217955 INFO  (jetty-launcher-3075-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51535}
   [junit4]   2> 2217955 ERROR (jetty-launcher-3075-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2217955 INFO  (jetty-launcher-3075-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 2217955 INFO  (jetty-launcher-3075-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2217955 INFO  (jetty-launcher-3075-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2217955 INFO  (jetty-launcher-3075-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-23T05:18:52.761Z
   [junit4]   2> 2217957 INFO  (jetty-launcher-3075-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@362880a{/solr,null,AVAILABLE}
   [junit4]   2> 2217958 INFO  (jetty-launcher-3075-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@769f5d00{SSL,[ssl, http/1.1]}{127.0.0.1:51536}
   [junit4]   2> 2217958 INFO  (jetty-launcher-3075-thread-2) [    ] o.e.j.s.Server Started @2223405ms
   [junit4]   2> 2217958 INFO  (jetty-launcher-3075-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51536}
   [junit4]   2> 2217958 ERROR (jetty-launcher-3075-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2217958 INFO  (jetty-launcher-3075-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 2217958 INFO  (jetty-launcher-3075-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2217959 INFO  (jetty-launcher-3075-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2217959 INFO  (jetty-launcher-3075-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-23T05:18:52.765Z
   [junit4]   2> 2217964 INFO  (jetty-launcher-3075-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2217967 INFO  (jetty-launcher-3075-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2217968 INFO  (jetty-launcher-3075-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2217999 INFO  (jetty-launcher-3075-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2218002 INFO  (jetty-launcher-3075-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2218003 INFO  (jetty-launcher-3075-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2218003 INFO  (jetty-launcher-3075-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51531/solr
   [junit4]   2> 2218004 INFO  (jetty-launcher-3075-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51531/solr
   [junit4]   2> 2218006 INFO  (jetty-launcher-3075-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2218007 INFO  (jetty-launcher-3075-thread-4) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2218007 INFO  (jetty-launcher-3075-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51531/solr
   [junit4]   2> 2218008 INFO  (jetty-launcher-3075-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51531/solr
   [junit4]   2> 2218054 INFO  (jetty-launcher-3075-thread-4) [n:127.0.0.1:51533_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2218055 INFO  (jetty-launcher-3075-thread-3) [n:127.0.0.1:51534_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2218056 INFO  (jetty-launcher-3075-thread-2) [n:127.0.0.1:51536_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2218056 INFO  (jetty-launcher-3075-thread-1) [n:127.0.0.1:51535_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2218057 INFO  (jetty-launcher-3075-thread-4) [n:127.0.0.1:51533_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:51533_solr
   [junit4]   2> 2218058 INFO  (jetty-launcher-3075-thread-3) [n:127.0.0.1:51534_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51534_solr
   [junit4]   2> 2218059 INFO  (jetty-launcher-3075-thread-2) [n:127.0.0.1:51536_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51536_solr
   [junit4]   2> 2218059 INFO  (jetty-launcher-3075-thread-1) [n:127.0.0.1:51535_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51535_solr
   [junit4]   2> 2218060 INFO  (jetty-launcher-3075-thread-4) [n:127.0.0.1:51533_solr    ] o.a.s.c.Overseer Overseer (id=97506238963122188-127.0.0.1:51533_solr-n_0000000000) starting
   [junit4]   2> 2218061 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2218061 INFO  (zkCallback-3094-thread-1-processing-n:127.0.0.1:51534_solr) [n:127.0.0.1:51534_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2218061 INFO  (zkCallback-3097-thread-1-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2218061 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2218064 INFO  (zkCallback-3097-thread-1-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2218066 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2218066 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2218066 INFO  (zkCallback-3094-thread-1-processing-n:127.0.0.1:51534_solr) [n:127.0.0.1:51534_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2218086 INFO  (jetty-launcher-3075-thread-4) [n:127.0.0.1:51533_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51533_solr
   [junit4]   2> 2218088 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2218088 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2218088 INFO  (zkCallback-3094-thread-1-processing-n:127.0.0.1:51534_solr) [n:127.0.0.1:51534_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2218089 INFO  (zkCallback-3097-thread-1-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2218222 INFO  (jetty-launcher-3075-thread-2) [n:127.0.0.1:51536_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node2/.
   [junit4]   2> 2218224 INFO  (jetty-launcher-3075-thread-1) [n:127.0.0.1:51535_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node1/.
   [junit4]   2> 2218226 INFO  (jetty-launcher-3075-thread-4) [n:127.0.0.1:51533_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node4/.
   [junit4]   2> 2218227 INFO  (jetty-launcher-3075-thread-3) [n:127.0.0.1:51534_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node3/.
   [junit4]   2> 2219940 INFO  (SUITE-CustomCollectionTest-seed#[D1E37C890235A44]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2219942 INFO  (SUITE-CustomCollectionTest-seed#[D1E37C890235A44]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:51531/solr ready
   [junit4]   2> 2219992 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[D1E37C890235A44]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateShardRepFactor
   [junit4]   2> 2219993 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[D1E37C890235A44]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2220022 INFO  (qtp432236786-17158) [n:127.0.0.1:51535_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=a,b&replicationFactor=1&collection.configName=conf&name=testCreateShardRepFactor&router.name=implicit&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2220026 INFO  (OverseerThreadFactory-8058-thread-1-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.CreateCollectionCmd Create collection testCreateShardRepFactor
   [junit4]   2> 2220157 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_a_replica1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=a&wt=javabin&version=2
   [junit4]   2> 2220176 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_b_replica1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=b&wt=javabin&version=2
   [junit4]   2> 2220283 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2220283 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2221177 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2221186 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_a_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2221190 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2221207 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_b_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2221217 WARN  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_a_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2221218 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2221220 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_a_replica1' using configuration from collection testCreateShardRepFactor
   [junit4]   2> 2221220 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SolrCore [[testCreateShardRepFactor_a_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node1/testCreateShardRepFactor_a_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node1/./testCreateShardRepFactor_a_replica1/data/]
   [junit4]   2> 2221226 WARN  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_b_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2221227 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2221228 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_b_replica1' using configuration from collection testCreateShardRepFactor
   [junit4]   2> 2221229 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SolrCore [[testCreateShardRepFactor_b_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node2/testCreateShardRepFactor_b_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node2/./testCreateShardRepFactor_b_replica1/data/]
   [junit4]   2> 2221299 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2221299 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2221299 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2221300 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2221302 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2221302 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2221302 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2221302 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2221302 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@28b8d152[testCreateShardRepFactor_a_replica1] main]
   [junit4]   2> 2221302 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@51a13d5d[testCreateShardRepFactor_b_replica1] main]
   [junit4]   2> 2221304 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2221304 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2221305 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2221305 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2221305 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2221305 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2221305 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560099827074727936
   [junit4]   2> 2221305 INFO  (searcherExecutor-8067-thread-1-processing-n:127.0.0.1:51535_solr x:testCreateShardRepFactor_a_replica1 s:a c:testCreateShardRepFactor) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SolrCore [testCreateShardRepFactor_a_replica1] Registered new searcher Searcher@28b8d152[testCreateShardRepFactor_a_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2221305 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560099827074727936
   [junit4]   2> 2221305 INFO  (searcherExecutor-8068-thread-1-processing-n:127.0.0.1:51536_solr x:testCreateShardRepFactor_b_replica1 s:b c:testCreateShardRepFactor) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SolrCore [testCreateShardRepFactor_b_replica1] Registered new searcher Searcher@51a13d5d[testCreateShardRepFactor_b_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2221316 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2221316 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2221316 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2221317 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:51536/solr/testCreateShardRepFactor_b_replica1/
   [junit4]   2> 2221316 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2221317 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2221317 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.SyncStrategy https://127.0.0.1:51536/solr/testCreateShardRepFactor_b_replica1/ has no replicas
   [junit4]   2> 2221317 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:51535/solr/testCreateShardRepFactor_a_replica1/
   [junit4]   2> 2221317 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2221317 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2221317 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.SyncStrategy https://127.0.0.1:51535/solr/testCreateShardRepFactor_a_replica1/ has no replicas
   [junit4]   2> 2221317 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2221325 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:51535/solr/testCreateShardRepFactor_a_replica1/ a
   [junit4]   2> 2221325 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:51536/solr/testCreateShardRepFactor_b_replica1/ b
   [junit4]   2> 2221437 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2221437 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2221491 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2221491 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2221495 INFO  (qtp763326128-17170) [n:127.0.0.1:51536_solr c:testCreateShardRepFactor s:b  x:testCreateShardRepFactor_b_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_b_replica1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=b&wt=javabin&version=2} status=0 QTime=1318
   [junit4]   2> 2221496 INFO  (qtp432236786-17163) [n:127.0.0.1:51535_solr c:testCreateShardRepFactor s:a  x:testCreateShardRepFactor_a_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=testCreateShardRepFactor_a_replica1&action=CREATE&numShards=2&collection=testCreateShardRepFactor&shard=a&wt=javabin&version=2} status=0 QTime=1338
   [junit4]   2> 2221500 INFO  (qtp432236786-17158) [n:127.0.0.1:51535_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2221609 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2221609 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2222034 INFO  (OverseerCollectionConfigSetProcessor-97506238963122188-127.0.0.1:51533_solr-n_0000000000) [n:127.0.0.1:51533_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2222501 INFO  (qtp432236786-17158) [n:127.0.0.1:51535_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={shards=a,b&replicationFactor=1&collection.configName=conf&name=testCreateShardRepFactor&router.name=implicit&action=CREATE&wt=javabin&version=2} status=0 QTime=2479
   [junit4]   2> 2222504 INFO  (qtp432236786-17154) [n:127.0.0.1:51535_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :createshard with params action=CREATESHARD&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2222507 INFO  (OverseerThreadFactory-8058-thread-2-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.CreateShardCmd Create shard invoked: {
   [junit4]   2>   "collection":"testCreateShardRepFactor",
   [junit4]   2>   "shard":"x",
   [junit4]   2>   "operation":"createshard"}
   [junit4]   2> 2222626 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2222626 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2222722 INFO  (OverseerThreadFactory-8058-thread-2-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.CreateShardCmd Creating shard testCreateShardRepFactor_x_replica1 as part of slice x of collection testCreateShardRepFactor on 127.0.0.1:51534_solr
   [junit4]   2> 2222744 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&name=testCreateShardRepFactor_x_replica1&action=CREATE&numShards=1&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2
   [junit4]   2> 2222862 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2222862 INFO  (zkCallback-3094-thread-1-processing-n:127.0.0.1:51534_solr) [n:127.0.0.1:51534_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2222862 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2223767 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2223775 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_x_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2223795 WARN  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.IndexSchema [testCreateShardRepFactor_x_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2223796 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2223798 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.CoreContainer Creating SolrCore 'testCreateShardRepFactor_x_replica1' using configuration from collection testCreateShardRepFactor
   [junit4]   2> 2223798 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SolrCore [[testCreateShardRepFactor_x_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node3/testCreateShardRepFactor_x_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node3/./testCreateShardRepFactor_x_replica1/data/]
   [junit4]   2> 2223869 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2223869 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2223871 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2223871 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2223871 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@34a55886[testCreateShardRepFactor_x_replica1] main]
   [junit4]   2> 2223873 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2223873 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2223873 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2223874 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560099829768519680
   [junit4]   2> 2223874 INFO  (searcherExecutor-8077-thread-1-processing-n:127.0.0.1:51534_solr x:testCreateShardRepFactor_x_replica1 s:x c:testCreateShardRepFactor) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SolrCore [testCreateShardRepFactor_x_replica1] Registered new searcher Searcher@34a55886[testCreateShardRepFactor_x_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2223883 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2223883 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2223884 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:51534/solr/testCreateShardRepFactor_x_replica1/
   [junit4]   2> 2223884 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2223884 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.SyncStrategy https://127.0.0.1:51534/solr/testCreateShardRepFactor_x_replica1/ has no replicas
   [junit4]   2> 2223884 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2223892 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:51534/solr/testCreateShardRepFactor_x_replica1/ x
   [junit4]   2> 2224002 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2224002 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2224002 INFO  (zkCallback-3094-thread-1-processing-n:127.0.0.1:51534_solr) [n:127.0.0.1:51534_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testCreateShardRepFactor/state.json] for collection [testCreateShardRepFactor] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2224010 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2224014 INFO  (qtp1663436973-17148) [n:127.0.0.1:51534_solr c:testCreateShardRepFactor s:x  x:testCreateShardRepFactor_x_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&name=testCreateShardRepFactor_x_replica1&action=CREATE&numShards=1&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2} status=0 QTime=1269
   [junit4]   2> 2224014 INFO  (OverseerThreadFactory-8058-thread-2-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.CreateShardCmd Finished create command on all shards for collection: testCreateShardRepFactor
   [junit4]   2> 2224017 INFO  (qtp432236786-17154) [n:127.0.0.1:51535_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CREATESHARD&collection=testCreateShardRepFactor&shard=x&wt=javabin&version=2} status=0 QTime=1513
   [junit4]   2> 2224019 INFO  (TEST-CustomCollectionTest.testCreateShardRepFactor-seed#[D1E37C890235A44]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateShardRepFactor
   [junit4]   2> 2224062 INFO  (TEST-CustomCollectionTest.testRouteFieldForImplicitRouter-seed#[D1E37C890235A44]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRouteFieldForImplicitRouter
   [junit4]   2> 2224063 INFO  (TEST-CustomCollectionTest.testRouteFieldForImplicitRouter-seed#[D1E37C890235A44]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 2224068 INFO  (qtp432236786-17156) [n:127.0.0.1:51535_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=testCreateShardRepFactor&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2224072 INFO  (OverseerCollectionConfigSetProcessor-97506238963122188-127.0.0.1:51533_solr-n_0000000000) [n:127.0.0.1:51533_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2224072 INFO  (OverseerThreadFactory-8058-thread-3-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 2224076 INFO  (qtp432236786-17157) [n:127.0.0.1:51535_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.testCreateShardRepFactor.a.replica1
   [junit4]   2> 2224076 INFO  (qtp763326128-17171) [n:127.0.0.1:51536_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.testCreateShardRepFactor.b.replica1
   [junit4]   2> 2224076 INFO  (qtp432236786-17157) [n:127.0.0.1:51535_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_a_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1aff3bb1
   [junit4]   2> 2224076 INFO  (qtp763326128-17171) [n:127.0.0.1:51536_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_b_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5733a3bd
   [junit4]   2> 2224081 INFO  (qtp432236786-17157) [n:127.0.0.1:51535_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.testCreateShardRepFactor.a.replica1
   [junit4]   2> 2224081 INFO  (qtp1663436973-17149) [n:127.0.0.1:51534_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.testCreateShardRepFactor.x.replica1
   [junit4]   2> 2224081 INFO  (qtp763326128-17171) [n:127.0.0.1:51536_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.testCreateShardRepFactor.b.replica1
   [junit4]   2> 2224081 INFO  (qtp1663436973-17149) [n:127.0.0.1:51534_solr    ] o.a.s.c.SolrCore [testCreateShardRepFactor_x_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@60b4410b
   [junit4]   2> 2224082 INFO  (qtp1663436973-17149) [n:127.0.0.1:51534_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.testCreateShardRepFactor.x.replica1
   [junit4]   2> 2224085 INFO  (qtp763326128-17171) [n:127.0.0.1:51536_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testCreateShardRepFactor_b_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=9
   [junit4]   2> 2224085 INFO  (qtp432236786-17157) [n:127.0.0.1:51535_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testCreateShardRepFactor_a_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=10
   [junit4]   2> 2224086 INFO  (qtp1663436973-17149) [n:127.0.0.1:51534_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=testCreateShardRepFactor_x_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=5
   [junit4]   2> 2224811 INFO  (qtp432236786-17156) [n:127.0.0.1:51535_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=testCreateShardRepFactor&action=DELETE&wt=javabin&version=2} status=0 QTime=743
   [junit4]   2> 2224815 INFO  (qtp432236786-17162) [n:127.0.0.1:51535_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=a,b,c,d&replicationFactor=2&collection.configName=conf&maxShardsPerNode=3&router.field=shard_s&name=withShardField&router.name=implicit&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2224818 INFO  (OverseerThreadFactory-8058-thread-4-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.CreateCollectionCmd Create collection withShardField
   [junit4]   2> 2224819 INFO  (OverseerCollectionConfigSetProcessor-97506238963122188-127.0.0.1:51533_solr-n_0000000000) [n:127.0.0.1:51533_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2225033 INFO  (qtp432236786-17160) [n:127.0.0.1:51535_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_c_replica2&action=CREATE&numShards=4&collection=withShardField&shard=c&wt=javabin&version=2
   [junit4]   2> 2225039 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_a_replica1&action=CREATE&numShards=4&collection=withShardField&shard=a&wt=javabin&version=2
   [junit4]   2> 2225046 INFO  (qtp432236786-17158) [n:127.0.0.1:51535_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_a_replica2&action=CREATE&numShards=4&collection=withShardField&shard=a&wt=javabin&version=2
   [junit4]   2> 2225046 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_d_replica1&action=CREATE&numShards=4&collection=withShardField&shard=d&wt=javabin&version=2
   [junit4]   2> 2225055 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_c_replica1&action=CREATE&numShards=4&collection=withShardField&shard=c&wt=javabin&version=2
   [junit4]   2> 2225056 INFO  (qtp1663436973-17146) [n:127.0.0.1:51534_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_b_replica1&action=CREATE&numShards=4&collection=withShardField&shard=b&wt=javabin&version=2
   [junit4]   2> 2225082 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_d_replica2&action=CREATE&numShards=4&collection=withShardField&shard=d&wt=javabin&version=2
   [junit4]   2> 2225087 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=withShardField_b_replica2&action=CREATE&numShards=4&collection=withShardField&shard=b&wt=javabin&version=2
   [junit4]   2> 2225221 INFO  (zkCallback-3097-thread-3-processing-n:127.0.0.1:51533_solr) [n:127.0.0.1:51533_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withShardField/state.json] for collection [withShardField] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2225222 INFO  (zkCallback-3094-thread-1-processing-n:127.0.0.1:51534_solr) [n:127.0.0.1:51534_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withShardField/state.json] for collection [withShardField] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2225221 INFO  (zkCallback-3095-thread-1-processing-n:127.0.0.1:51536_solr) [n:127.0.0.1:51536_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withShardField/state.json] for collection [withShardField] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2225221 INFO  (zkCallback-3096-thread-1-processing-n:127.0.0.1:51535_solr) [n:127.0.0.1:51535_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/withShardField/state.json] for collection [withShardField] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 2226091 INFO  (qtp432236786-17158) [n:127.0.0.1:51535_solr c:withShardField s:a  x:withShardField_a_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2226091 INFO  (qtp432236786-17160) [n:127.0.0.1:51535_solr c:withShardField s:c  x:withShardField_c_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2226096 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2226130 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2226130 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2226133 INFO  (qtp1663436973-17146) [n:127.0.0.1:51534_solr c:withShardField s:b  x:withShardField_b_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2226140 INFO  (qtp432236786-17158) [n:127.0.0.1:51535_solr c:withShardField s:a  x:withShardField_a_replica2] o.a.s.s.IndexSchema [withShardField_a_replica2] Schema name=cloud-dynamic
   [junit4]   2> 2226141 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.s.IndexSchema [withShardField_d_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2226141 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.s.IndexSchema [withShardField_a_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2226169 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.s.IndexSchema [withShardField_c_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2226181 INFO  (qtp1663436973-17146) [n:127.0.0.1:51534_solr c:withShardField s:b  x:withShardField_b_replica1] o.a.s.s.IndexSchema [withShardField_b_replica1] Schema name=cloud-dynamic
   [junit4]   2> 2226202 WARN  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.s.IndexSchema [withShardField_c_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2226204 WARN  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.s.IndexSchema [withShardField_a_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2226204 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2226204 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 2226205 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2226208 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_a_replica1' using configuration from collection withShardField
   [junit4]   2> 2226209 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.c.SolrCore [[withShardField_a_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node2/withShardField_a_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node2/./withShardField_a_replica1/data/]
   [junit4]   2> 2226226 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.s.IndexSchema [withShardField_d_replica2] Schema name=cloud-dynamic
   [junit4]   2> 2226226 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.s.IndexSchema [withShardField_b_replica2] Schema name=cloud-dynamic
   [junit4]   2> 2226232 INFO  (qtp432236786-17160) [n:127.0.0.1:51535_solr c:withShardField s:c  x:withShardField_c_replica2] o.a.s.s.IndexSchema [withShardField_c_replica2] Schema name=cloud-dynamic
   [junit4]   2> 2226255 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2226256 WARN  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.s.IndexSchema [withShardField_b_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2226256 WARN  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.s.IndexSchema [withShardField_d_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2226257 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2226257 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2226257 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_b_replica2' using configuration from collection withShardField
   [junit4]   2> 2226258 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_d_replica2' using configuration from collection withShardField
   [junit4]   2> 2226258 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.c.SolrCore [[withShardField_b_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node4/withShardField_b_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node4/./withShardField_b_replica2/data/]
   [junit4]   2> 2226258 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.c.SolrCore [[withShardField_d_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node4/withShardField_d_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node4/./withShardField_d_replica2/data/]
   [junit4]   2> 2226273 WARN  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.s.IndexSchema [withShardField_d_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2226274 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2226276 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_d_replica1' using configuration from collection withShardField
   [junit4]   2> 2226276 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.c.SolrCore [[withShardField_d_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node3/withShardField_d_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node3/./withShardField_d_replica1/data/]
   [junit4]   2> 2226276 WARN  (qtp1663436973-17146) [n:127.0.0.1:51534_solr c:withShardField s:b  x:withShardField_b_replica1] o.a.s.s.IndexSchema [withShardField_b_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2226277 INFO  (qtp1663436973-17146) [n:127.0.0.1:51534_solr c:withShardField s:b  x:withShardField_b_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2226278 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_c_replica1' using configuration from collection withShardField
   [junit4]   2> 2226278 INFO  (qtp1663436973-17146) [n:127.0.0.1:51534_solr c:withShardField s:b  x:withShardField_b_replica1] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_b_replica1' using configuration from collection withShardField
   [junit4]   2> 2226278 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.c.SolrCore [[withShardField_c_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node2/withShardField_c_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node2/./withShardField_c_replica1/data/]
   [junit4]   2> 2226278 WARN  (qtp432236786-17158) [n:127.0.0.1:51535_solr c:withShardField s:a  x:withShardField_a_replica2] o.a.s.s.IndexSchema [withShardField_a_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2226278 INFO  (qtp1663436973-17146) [n:127.0.0.1:51534_solr c:withShardField s:b  x:withShardField_b_replica1] o.a.s.c.SolrCore [[withShardField_b_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node3/withShardField_b_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node3/./withShardField_b_replica1/data/]
   [junit4]   2> 2226279 INFO  (qtp432236786-17158) [n:127.0.0.1:51535_solr c:withShardField s:a  x:withShardField_a_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2226279 WARN  (qtp432236786-17160) [n:127.0.0.1:51535_solr c:withShardField s:c  x:withShardField_c_replica2] o.a.s.s.IndexSchema [withShardField_c_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2226280 INFO  (qtp432236786-17160) [n:127.0.0.1:51535_solr c:withShardField s:c  x:withShardField_c_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 2226280 INFO  (qtp432236786-17158) [n:127.0.0.1:51535_solr c:withShardField s:a  x:withShardField_a_replica2] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_a_replica2' using configuration from collection withShardField
   [junit4]   2> 2226281 INFO  (qtp432236786-17158) [n:127.0.0.1:51535_solr c:withShardField s:a  x:withShardField_a_replica2] o.a.s.c.SolrCore [[withShardField_a_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node1/withShardField_a_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node1/./withShardField_a_replica2/data/]
   [junit4]   2> 2226281 INFO  (qtp432236786-17160) [n:127.0.0.1:51535_solr c:withShardField s:c  x:withShardField_c_replica2] o.a.s.c.CoreContainer Creating SolrCore 'withShardField_c_replica2' using configuration from collection withShardField
   [junit4]   2> 2226281 INFO  (qtp432236786-17160) [n:127.0.0.1:51535_solr c:withShardField s:c  x:withShardField_c_replica2] o.a.s.c.SolrCore [[withShardField_c_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node1/withShardField_c_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_D1E37C890235A44-001/tempDir-001/node1/./withShardField_c_replica2/data/]
   [junit4]   2> 2226373 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2226374 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2226392 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2226392 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2226394 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2226394 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2226398 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2226398 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2226404 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@487f4986[withShardField_b_replica2] main]
   [junit4]   2> 2226404 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1b5c981c[withShardField_a_replica1] main]
   [junit4]   2> 2226406 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2226407 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2226407 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2226407 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2226407 INFO  (searcherExecutor-8083-thread-1-processing-n:127.0.0.1:51533_solr x:withShardField_b_replica2 s:b c:withShardField) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.c.SolrCore [withShardField_b_replica2] Registered new searcher Searcher@487f4986[withShardField_b_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2226407 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560099832424562688
   [junit4]   2> 2226407 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2226407 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2226407 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560099832424562688
   [junit4]   2> 2226408 INFO  (searcherExecutor-8082-thread-1-processing-n:127.0.0.1:51536_solr x:withShardField_a_replica1 s:a c:withShardField) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.c.SolrCore [withShardField_a_replica1] Registered new searcher Searcher@1b5c981c[withShardField_a_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2226416 INFO  (qtp848927219-17141) [n:127.0.0.1:51533_solr c:withShardField s:b  x:withShardField_b_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard b: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2226423 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2226423 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2226425 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2226425 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2226430 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2226430 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2226432 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2226433 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2226436 INFO  (qtp763326128-17168) [n:127.0.0.1:51536_solr c:withShardField s:a  x:withShardField_a_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard a: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2226450 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@23005a4b[withShardField_d_replica2] main]
   [junit4]   2> 2226453 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2226453 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2226454 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2226454 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560099832473845760
   [junit4]   2> 2226454 INFO  (searcherExecutor-8084-thread-1-processing-n:127.0.0.1:51533_solr x:withShardField_d_replica2 s:d c:withShardField) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.c.SolrCore [withShardField_d_replica2] Registered new searcher Searcher@23005a4b[withShardField_d_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2226459 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5a5967e0[withShardField_d_replica1] main]
   [junit4]   2> 2226470 INFO  (qtp848927219-17140) [n:127.0.0.1:51533_solr c:withShardField s:d  x:withShardField_d_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard d: total=2 found=1 timeoutin=9996ms
   [junit4]   2> 2226471 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 2226471 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2226471 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2226475 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2226475 INFO  (qtp763326128-17169) [n:127.0.0.1:51536_solr c:withShardField s:c  x:withShardField_c_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2226477 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2226477 INFO  (qtp1663436973-17147) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShardField_d_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2226477 INFO  (searcherExecutor-8085-thread-1-processing-n:127.0.0.1:51534_solr x:withShardField_d_replica1 s:d c:withShardField) [n:127.0.0.1:51534_solr c:withShardField s:d  x:withShard

[...truncated too long message...]

]   2> 2694045 INFO  (jetty-closer-3174-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2694045 INFO  (jetty-closer-3174-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1e7ce1b1{/solr,null,UNAVAILABLE}
   [junit4]   2> 2694045 INFO  (jetty-closer-3174-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1d78f1da{/solr,null,UNAVAILABLE}
   [junit4]   2> 2694044 INFO  (jetty-closer-3174-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1a5893db{/solr,null,UNAVAILABLE}
   [junit4]   2> 2694045 INFO  (jetty-closer-3174-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1ba263a7{/solr,null,UNAVAILABLE}
   [junit4]   2> 2694050 INFO  (SUITE-TestSegmentSorting-seed#[D1E37C890235A44]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53054 53054
   [junit4]   2> 2694058 INFO  (Thread-3932) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53054 53054
   [junit4]   2> 2699872 WARN  (Thread-3932) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	8	/solr/aliases.json
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	5	/solr/configs/class org.apache.solr.cloud.TestSegmentSorting_configSet
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	8	/solr/clusterstate.json
   [junit4]   2> 	8	/solr/clusterprops.json
   [junit4]   2> 	4	/solr/collections/testAtomicUpdateOfSegmentSortField/state.json
   [junit4]   2> 	2	/solr/collections/testSegmentTerminateEarly/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	41	/solr/overseer/collection-queue-work
   [junit4]   2> 	25	/solr/overseer/queue
   [junit4]   2> 	11	/solr/collections
   [junit4]   2> 	8	/solr/live_nodes
   [junit4]   2> 	6	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 2699872 INFO  (SUITE-TestSegmentSorting-seed#[D1E37C890235A44]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestSegmentSorting_D1E37C890235A44-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=996, maxMBSortInHeap=6.752229258357216, sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=ar-JO, timezone=Antarctica/Troll
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=237887120,total=527433728
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollationFieldDocValues, DebugComponentTest, TestCloudInspectUtil, SimpleCollectionCreateDeleteTest, TestSolrCoreSnapshots, DeleteStatusTest, CollectionStateFormat2Test, TestQuerySenderListener, TestMacroExpander, TestFuzzyAnalyzedSuggestions, TestReloadDeadlock, NoCacheHeaderTest, SearchHandlerTest, OverriddenZkACLAndCredentialsProvidersTest, TestPushWriter, AssignTest, OverseerModifyCollectionTest, TestDynamicFieldResource, TestMergePolicyConfig, TestSolrCoreProperties, SolrCloudExampleTest, TestOverriddenPrefixQueryForCustomFieldType, MultiThreadedOCPTest, TestPathTrie, SuggesterTSTTest, TestPerFieldSimilarityClassic, CloudExitableDirectoryReaderTest, HdfsLockFactoryTest, FieldAnalysisRequestHandlerTest, TestReload, TestSchemaVersionResource, SpellCheckComponentTest, SharedFSAutoReplicaFailoverTest, ScriptEngineTest, TestTolerantUpdateProcessorCloud, TestCollapseQParserPlugin, HdfsNNFailoverTest, HdfsRecoveryZkTest, TestJettySolrRunner, TestReRankQParserPlugin, RequestHandlersTest, SaslZkACLProviderTest, SolrGangliaReporterTest, TestUnifiedSolrHighlighter, TestReloadAndDeleteDocs, TestBlendedInfixSuggestions, BaseCdcrDistributedZkTest, BlockJoinFacetSimpleTest, TestRequestForwarding, JavabinLoaderTest, JsonLoaderTest, PKIAuthenticationIntegrationTest, TestStressUserVersions, ConnectionManagerTest, CurrencyFieldOpenExchangeTest, TestDocSet, HdfsSyncSliceTest, RemoteQueryErrorTest, TestManagedSynonymFilterFactory, MigrateRouteKeyTest, BufferStoreTest, HdfsChaosMonkeySafeLeaderTest, TestSolr4Spatial2, TestShortCircuitedRequests, TestConfigsApi, TestStressInPlaceUpdates, TestDefaultSearchFieldResource, TestReqParamsAPI, TestOmitPositions, DistributedFacetExistsSmallTest, CoreMergeIndexesAdminHandlerTest, SolrJmxReporterTest, TestUtils, TestCoreDiscovery, CacheHeaderTest, CachingDirectoryFactoryTest, DirectUpdateHandlerOptimizeTest, LoggingHandlerTest, TestFieldCache, MetricsHandlerTest, TestConfigSetsAPIZkFailure, NotRequiredUniqueKeyTest, TestBlobHandler, TestRawTransformer, TestManagedSchemaAPI, UpdateParamsTest, TestBackupRepositoryFactory, InfixSuggestersTest, TestNamedUpdateProcessors, TestSearchPerf, SimpleMLTQParserTest, TestPointFields, SuggestComponentContextFilterQueryTest, UpdateLogTest, UUIDFieldTest, TestQuerySenderNoQuery, TestZkChroot, ConcurrentDeleteAndCreateCollectionTest, TestSolrFieldCacheMBean, TestRestManager, OpenCloseCoreStressTest, SolrIndexConfigTest, PreAnalyzedFieldTest, TestCrossCoreJoin, TestConfigSetsAPI, TestLocalFSCloudBackupRestore, TestWriterPerf, TestLMJelinekMercerSimilarityFactory, EnumFieldTest, TestConfigSetImmutable, TestLeaderElectionZkExpiry, AddSchemaFieldsUpdateProcessorFactoryTest, DistributedFacetPivotSmallTest, MinimalSchemaTest, SuggesterTest, TestSchemaResource, TestGraphMLResponseWriter, ShufflingReplicaListTransformerTest, TestRealTimeGet, CopyFieldTest, RollingRestartTest, TestFastWriter, BasicDistributedZk2Test, SolrInfoMBeanTest, TestFieldCacheSanityChecker, BitVectorTest, LeaderInitiatedRecoveryOnCommitTest, AutoCommitTest, TestClusterProperties, BadIndexSchemaTest, TestSystemCollAutoCreate, TestSolrQueryParser, DeleteLastCustomShardedReplicaTest, TestConfig, TestBulkSchemaConcurrent, HttpSolrCallGetCoreTest, JsonValidatorTest, FileBasedSpellCheckerTest, TestRangeQuery, TestDFISimilarityFactory, HdfsTlogReplayBufferedWhileIndexingTest, TestUseDocValuesAsStored2, ClusterStateTest, TestConfigSets, TestIndexingPerformance, HttpPartitionTest, DistribDocExpirationUpdateProcessorTest, ResourceLoaderTest, SolrCmdDistributorTest, BlobRepositoryCloudTest, TestAtomicUpdateErrorCases, TestImplicitCoreProperties, TestSha256AuthenticationProvider, LeaderFailoverAfterPartitionTest, SignatureUpdateProcessorFactoryTest, TestFieldResource, DistributedFacetPivotLongTailTest, TestClusterStateMutator, ShowFileRequestHandlerTest, TestFreeTextSuggestions, TestSolr4Spatial, WordBreakSolrSpellCheckerTest, TestSolrCloudWithSecureImpersonation, TestCustomStream, TestSchemalessBufferedUpdates, FullSolrCloudDistribCmdsTest, TestReversedWildcardFilterFactory, BlockCacheTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, LeaderElectionIntegrationTest, BasicZkTest, TestRandomFaceting, ZkSolrClientTest, ZkCLITest, ShardRoutingCustomTest, TestDistributedGrouping, TestRecovery, TestHashPartitioner, TermVectorComponentDistributedTest, TestStressReorder, TestJoin, DistributedTermsComponentTest, TestCoreContainer, SolrCoreTest, StatsComponentTest, ConvertedLegacyTest, TestSort, TestFiltering, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores, DirectUpdateHandlerTest, TestBadConfig, TestIndexSearcher, HighlighterTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SpellCheckCollatorTest, CoreAdminHandlerTest, TestFoldingMultitermQuery, DocValuesTest, TestStressLucene, TestTrie, SuggesterWFSTTest, PolyFieldTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, QueryEqualityTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, DocValuesMultiTest, DistanceFunctionTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, LukeRequestHandlerTest, TestQueryUtils, DocumentAnalysisRequestHandlerTest, TestQueryTypes, PrimitiveFieldTypeTest, XmlUpdateRequestHandlerTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, RequiredFieldsTest, FastVectorHighlighterTest, RegexBoostProcessorTest, IndexSchemaTest, TestJmxIntegration, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, MBeansHandlerTest, QueryParsingTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, HighlighterConfigTest, TestSolrIndexConfig, BadComponentTest, MultiTermTest, SampleTest, OutputWriterTest, ExternalFileFieldSortTest, TestPhraseSuggestions, SynonymTokenizerTest, TestXIncludeConfig, EchoParamsTest, TestSweetSpotSimilarityFactory, TestLMDirichletSimilarityFactory, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache, DateMathParserTest, PrimUtilsTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, TestLRUCache, SystemInfoHandlerTest, CircularListTest, CursorPagingTest, SolrTestCaseJ4Test, TestDistributedMissingSort, TestHighlightDedupGrouping, TestTolerantSearch, TestEmbeddedSolrServerConstructors, ConnectionReuseTest, ActionThrottleTest, CdcrBootstrapTest, CollectionsAPIAsyncDistributedZkTest, DeleteNodeTest, DeleteReplicaTest, DistribCursorPagingTest, MissingSegmentRecoveryTest, OverseerTaskQueueTest, ReplicationFactorTest, TestAuthenticationFramework, TestCloudPseudoReturnFields, TestCloudRecovery, TestDownShardTolerantSearch, TestHdfsCloudBackupRestore, TestLeaderElectionWithEmptyReplica, TestMiniSolrCloudClusterSSL, TestRandomRequestDistribution, TestSSLRandomization, TestSegmentSorting]
   [junit4] Completed [531/696 (2!)] on J1 in 61.68s, 2 tests, 2 failures <<< FAILURES!

[...truncated 63253 lines...]


[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 714 - Still unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/714/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

6 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
expected:<0> but was:<4>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<4>
	at __randomizedtesting.SeedInfo.seed([144CBF0BFE38B622:9C1880D150C4DBDA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:146)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter

Error Message:
Error from server at http://127.0.0.1:59115/solr/withShardField: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /solr/withShardField/update. Reason: <pre>    java.lang.NoClassDefFoundError: org/apache/solr/cloud/LeaderInitiatedRecoveryThread</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:59115/solr/withShardField: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /solr/withShardField/update. Reason:
<pre>    java.lang.NoClassDefFoundError: org/apache/solr/cloud/LeaderInitiatedRecoveryThread</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([144CBF0BFE38B622:411C579952C179D2]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:578)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1358)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
	at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:232)
	at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:141)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
Error from server at http://127.0.0.1:62537/c_i/ch/forceleader_test_collection_shard1_replica3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /c_i/ch/forceleader_test_collection_shard1_replica3/update. Reason: <pre>    java.lang.NoClassDefFoundError: org/apache/solr/cloud/LeaderInitiatedRecoveryThread</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:62537/c_i/ch/forceleader_test_collection_shard1_replica3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /c_i/ch/forceleader_test_collection_shard1_replica3/update. Reason:
<pre>    java.lang.NoClassDefFoundError: org/apache/solr/cloud/LeaderInitiatedRecoveryThread</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([144CBF0BFE38B622:F2DB8BCBC7BA4F43]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:803)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1238)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
	at org.apache.solr.cloud.ForceLeaderTest.sendDoc(ForceLeaderTest.java:424)
	at org.apache.solr.cloud.ForceLeaderTest.putNonLeadersIntoLIR(ForceLeaderTest.java:337)
	at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:90)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:62537/c_i/ch/forceleader_test_collection_shard1_replica3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /c_i/ch/forceleader_test_collection_shard1_replica3/update. Reason:
<pre>    java.lang.NoClassDefFoundError: org/apache/solr/cloud/LeaderInitiatedRecoveryThread</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:578)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:780)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Error from server at http://127.0.0.1:62671/w_pi/pq/c8n_1x3_lf_shard1_replica1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /w_pi/pq/c8n_1x3_lf_shard1_replica1/update. Reason: <pre>    java.lang.NoClassDefFoundError: org/apache/solr/cloud/LeaderInitiatedRecoveryThread</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:62671/w_pi/pq/c8n_1x3_lf_shard1_replica1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /w_pi/pq/c8n_1x3_lf_shard1_replica1/update. Reason:
<pre>    java.lang.NoClassDefFoundError: org/apache/solr/cloud/LeaderInitiatedRecoveryThread</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([144CBF0BFE38B622:9C1880D150C4DBDA]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:578)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:813)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1238)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.HttpPartitionTest.sendDoc(HttpPartitionTest.java:609)
	at org.apache.solr.cloud.HttpPartitionTest.sendDoc(HttpPartitionTest.java:595)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:82)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestSegmentSorting.testAtomicUpdateOfSegmentSortField

Error Message:
Could not find a healthy node to handle the request.

Stack Trace:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.
	at __randomizedtesting.SeedInfo.seed([144CBF0BFE38B622:F55E980E287D2D9A]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1352)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1212)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
	at org.apache.solr.cloud.TestSegmentSorting.testAtomicUpdateOfSegmentSortField(TestSegmentSorting.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestSegmentSorting.testSegmentTerminateEarly

Error Message:
Could not find collection:testSegmentTerminateEarly

Stack Trace:
java.lang.AssertionError: Could not find collection:testSegmentTerminateEarly
	at __randomizedtesting.SeedInfo.seed([144CBF0BFE38B622:C4EA78A6226A2248]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
	at org.apache.solr.cloud.TestSegmentSorting.createCollection(TestSegmentSorting.java:89)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:941)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11656 lines...]
   [junit4] Suite: org.apache.solr.cloud.CustomCollectionTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/init-core-data-001
   [junit4]   2> 1070107 INFO  (SUITE-CustomCollectionTest-seed#[144CBF0BFE38B622]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1070112 INFO  (SUITE-CustomCollectionTest-seed#[144CBF0BFE38B622]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1070113 INFO  (SUITE-CustomCollectionTest-seed#[144CBF0BFE38B622]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001
   [junit4]   2> 1070113 INFO  (SUITE-CustomCollectionTest-seed#[144CBF0BFE38B622]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1070113 INFO  (Thread-1897) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1070113 INFO  (Thread-1897) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1070223 INFO  (SUITE-CustomCollectionTest-seed#[144CBF0BFE38B622]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:59112
   [junit4]   2> 1070246 INFO  (jetty-launcher-2291-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1070246 INFO  (jetty-launcher-2291-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1070246 INFO  (jetty-launcher-2291-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1070247 INFO  (jetty-launcher-2291-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1070253 INFO  (jetty-launcher-2291-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@553a2ff6{/solr,null,AVAILABLE}
   [junit4]   2> 1070254 INFO  (jetty-launcher-2291-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@282c8380{HTTP/1.1,[http/1.1]}{127.0.0.1:59114}
   [junit4]   2> 1070254 INFO  (jetty-launcher-2291-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f79239e{/solr,null,AVAILABLE}
   [junit4]   2> 1070254 INFO  (jetty-launcher-2291-thread-3) [    ] o.e.j.s.Server Started @1075438ms
   [junit4]   2> 1070254 INFO  (jetty-launcher-2291-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59114}
   [junit4]   2> 1070254 INFO  (jetty-launcher-2291-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@482b3875{/solr,null,AVAILABLE}
   [junit4]   2> 1070254 ERROR (jetty-launcher-2291-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1070254 INFO  (jetty-launcher-2291-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2e2b606a{HTTP/1.1,[http/1.1]}{127.0.0.1:59115}
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-4) [    ] o.e.j.s.Server Started @1075438ms
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59115}
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1070255 ERROR (jetty-launcher-2291-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-22T19:03:29.195Z
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7f22641e{/solr,null,AVAILABLE}
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1070255 INFO  (jetty-launcher-2291-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-22T19:03:29.195Z
   [junit4]   2> 1070256 INFO  (jetty-launcher-2291-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@88a1a{HTTP/1.1,[http/1.1]}{127.0.0.1:59117}
   [junit4]   2> 1070256 INFO  (jetty-launcher-2291-thread-2) [    ] o.e.j.s.Server Started @1075439ms
   [junit4]   2> 1070256 INFO  (jetty-launcher-2291-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59117}
   [junit4]   2> 1070256 ERROR (jetty-launcher-2291-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1070256 INFO  (jetty-launcher-2291-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 1070256 INFO  (jetty-launcher-2291-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1070256 INFO  (jetty-launcher-2291-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1070256 INFO  (jetty-launcher-2291-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@658d0db4{HTTP/1.1,[http/1.1]}{127.0.0.1:59116}
   [junit4]   2> 1070257 INFO  (jetty-launcher-2291-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-22T19:03:29.196Z
   [junit4]   2> 1070257 INFO  (jetty-launcher-2291-thread-1) [    ] o.e.j.s.Server Started @1075440ms
   [junit4]   2> 1070257 INFO  (jetty-launcher-2291-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59116}
   [junit4]   2> 1070257 ERROR (jetty-launcher-2291-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1070257 INFO  (jetty-launcher-2291-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 1070257 INFO  (jetty-launcher-2291-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1070258 INFO  (jetty-launcher-2291-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1070258 INFO  (jetty-launcher-2291-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-22T19:03:29.198Z
   [junit4]   2> 1070262 INFO  (jetty-launcher-2291-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1070262 INFO  (jetty-launcher-2291-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1070263 INFO  (jetty-launcher-2291-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1070263 INFO  (jetty-launcher-2291-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1070281 INFO  (jetty-launcher-2291-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1070282 INFO  (jetty-launcher-2291-thread-4) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1070282 INFO  (jetty-launcher-2291-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1070283 INFO  (jetty-launcher-2291-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1070283 INFO  (jetty-launcher-2291-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59112/solr
   [junit4]   2> 1070284 INFO  (jetty-launcher-2291-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59112/solr
   [junit4]   2> 1070286 INFO  (jetty-launcher-2291-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59112/solr
   [junit4]   2> 1070287 INFO  (jetty-launcher-2291-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59112/solr
   [junit4]   2> 1070364 INFO  (jetty-launcher-2291-thread-4) [n:127.0.0.1:59115_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1070364 INFO  (jetty-launcher-2291-thread-3) [n:127.0.0.1:59114_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1070365 INFO  (jetty-launcher-2291-thread-2) [n:127.0.0.1:59117_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1070366 INFO  (jetty-launcher-2291-thread-3) [n:127.0.0.1:59114_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59114_solr
   [junit4]   2> 1070366 INFO  (jetty-launcher-2291-thread-1) [n:127.0.0.1:59116_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1070367 INFO  (jetty-launcher-2291-thread-4) [n:127.0.0.1:59115_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59115_solr
   [junit4]   2> 1070368 INFO  (jetty-launcher-2291-thread-2) [n:127.0.0.1:59117_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59117_solr
   [junit4]   2> 1070369 INFO  (jetty-launcher-2291-thread-3) [n:127.0.0.1:59114_solr    ] o.a.s.c.Overseer Overseer (id=97503819140694026-127.0.0.1:59114_solr-n_0000000000) starting
   [junit4]   2> 1070369 INFO  (jetty-launcher-2291-thread-1) [n:127.0.0.1:59116_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59116_solr
   [junit4]   2> 1070371 INFO  (zkCallback-2310-thread-1-processing-n:127.0.0.1:59114_solr) [n:127.0.0.1:59114_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1070371 INFO  (zkCallback-2309-thread-1-processing-n:127.0.0.1:59116_solr) [n:127.0.0.1:59116_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1070372 INFO  (zkCallback-2311-thread-1-processing-n:127.0.0.1:59117_solr) [n:127.0.0.1:59117_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1070373 INFO  (zkCallback-2312-thread-1-processing-n:127.0.0.1:59115_solr) [n:127.0.0.1:59115_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1070375 INFO  (zkCallback-2311-thread-1-processing-n:127.0.0.1:59117_solr) [n:127.0.0.1:59117_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1070375 INFO  (zkCallback-2312-thread-1-processing-n:127.0.0.1:59115_solr) [n:127.0.0.1:59115_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1070375 INFO  (zkCallback-2309-thread-1-processing-n:127.0.0.1:59116_solr) [n:127.0.0.1:59116_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1070376 INFO  (zkCallback-2310-thread-1-processing-n:127.0.0.1:59114_solr) [n:127.0.0.1:59114_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1070379 INFO  (zkCallback-2311-thread-1-processing-n:127.0.0.1:59117_solr) [n:127.0.0.1:59117_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1070379 INFO  (zkCallback-2312-thread-1-processing-n:127.0.0.1:59115_solr) [n:127.0.0.1:59115_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1070379 INFO  (zkCallback-2309-thread-1-processing-n:127.0.0.1:59116_solr) [n:127.0.0.1:59116_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1070379 INFO  (zkCallback-2310-thread-1-processing-n:127.0.0.1:59114_solr) [n:127.0.0.1:59114_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1070403 INFO  (jetty-launcher-2291-thread-3) [n:127.0.0.1:59114_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59114_solr
   [junit4]   2> 1070407 INFO  (zkCallback-2311-thread-1-processing-n:127.0.0.1:59117_solr) [n:127.0.0.1:59117_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1070407 INFO  (zkCallback-2312-thread-1-processing-n:127.0.0.1:59115_solr) [n:127.0.0.1:59115_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1070407 INFO  (zkCallback-2309-thread-1-processing-n:127.0.0.1:59116_solr) [n:127.0.0.1:59116_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1070407 INFO  (zkCallback-2310-thread-1-processing-n:127.0.0.1:59114_solr) [n:127.0.0.1:59114_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1070545 INFO  (jetty-launcher-2291-thread-2) [n:127.0.0.1:59117_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node2/.
   [junit4]   2> 1070547 INFO  (jetty-launcher-2291-thread-1) [n:127.0.0.1:59116_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node1/.
   [junit4]   2> 1070551 INFO  (jetty-launcher-2291-thread-3) [n:127.0.0.1:59114_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node3/.
   [junit4]   2> 1070555 INFO  (jetty-launcher-2291-thread-4) [n:127.0.0.1:59115_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node4/.
   [junit4]   2> 1071536 INFO  (SUITE-CustomCollectionTest-seed#[144CBF0BFE38B622]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1071538 INFO  (SUITE-CustomCollectionTest-seed#[144CBF0BFE38B622]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59112/solr ready
   [junit4]   2> 1071591 INFO  (TEST-CustomCollectionTest.testRouteFieldForHashRouter-seed#[144CBF0BFE38B622]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRouteFieldForHashRouter
   [junit4]   2> 1071595 INFO  (TEST-CustomCollectionTest.testRouteFieldForHashRouter-seed#[144CBF0BFE38B622]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1071603 INFO  (qtp609904077-10342) [n:127.0.0.1:59115_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=3&router.field=shard_s&name=routeFieldColl&action=CREATE&numShards=4&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1071607 INFO  (OverseerThreadFactory-4371-thread-1-processing-n:127.0.0.1:59114_solr) [n:127.0.0.1:59114_solr    ] o.a.s.c.CreateCollectionCmd Create collection routeFieldColl
   [junit4]   2> 1071742 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard4_replica1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard4&wt=javabin&version=2
   [junit4]   2> 1071742 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard1_replica2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1071744 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard2_replica1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1071746 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard4_replica2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard4&wt=javabin&version=2
   [junit4]   2> 1071747 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard1_replica1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1071748 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard2_replica2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1071750 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard3_replica1&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard3&wt=javabin&version=2
   [junit4]   2> 1071751 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=routeFieldColl_shard3_replica2&action=CREATE&numShards=4&collection=routeFieldColl&shard=shard3&wt=javabin&version=2
   [junit4]   2> 1071899 INFO  (zkCallback-2311-thread-1-processing-n:127.0.0.1:59117_solr) [n:127.0.0.1:59117_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1071899 INFO  (zkCallback-2312-thread-1-processing-n:127.0.0.1:59115_solr) [n:127.0.0.1:59115_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1071899 INFO  (zkCallback-2310-thread-3-processing-n:127.0.0.1:59114_solr) [n:127.0.0.1:59114_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1071899 INFO  (zkCallback-2309-thread-1-processing-n:127.0.0.1:59116_solr) [n:127.0.0.1:59116_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/routeFieldColl/state.json] for collection [routeFieldColl] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1072816 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1072818 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1072818 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1072826 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1072827 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1072828 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1072847 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1072850 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1072859 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica2] Schema name=cloud-dynamic
   [junit4]   2> 1072859 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica1] Schema name=cloud-dynamic
   [junit4]   2> 1072860 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica1] Schema name=cloud-dynamic
   [junit4]   2> 1072873 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica1] Schema name=cloud-dynamic
   [junit4]   2> 1072873 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica2] Schema name=cloud-dynamic
   [junit4]   2> 1072901 WARN  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1072902 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1072904 WARN  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1072905 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica1] Schema name=cloud-dynamic
   [junit4]   2> 1072908 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1072927 WARN  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1072931 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1072937 WARN  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard4_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1072938 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1072939 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica2] Schema name=cloud-dynamic
   [junit4]   2> 1072950 WARN  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1072958 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard1_replica1' using configuration from collection routeFieldColl
   [junit4]   2> 1072959 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.SolrCore [[routeFieldColl_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node3/routeFieldColl_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node3/./routeFieldColl_shard1_replica1/data/]
   [junit4]   2> 1072960 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica2] Schema name=cloud-dynamic
   [junit4]   2> 1072960 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard2_replica1' using configuration from collection routeFieldColl
   [junit4]   2> 1072960 WARN  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard3_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1072961 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1072961 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.c.SolrCore [[routeFieldColl_shard2_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node4/routeFieldColl_shard2_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node4/./routeFieldColl_shard2_replica1/data/]
   [junit4]   2> 1072962 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard4_replica1' using configuration from collection routeFieldColl
   [junit4]   2> 1072962 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard3_replica2' using configuration from collection routeFieldColl
   [junit4]   2> 1072970 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.c.SolrCore [[routeFieldColl_shard4_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node4/routeFieldColl_shard4_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node4/./routeFieldColl_shard4_replica1/data/]
   [junit4]   2> 1072971 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.SolrCore [[routeFieldColl_shard3_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node1/routeFieldColl_shard3_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node1/./routeFieldColl_shard3_replica2/data/]
   [junit4]   2> 1072974 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard4_replica2' using configuration from collection routeFieldColl
   [junit4]   2> 1072974 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1072975 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.SolrCore [[routeFieldColl_shard4_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node2/routeFieldColl_shard4_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node2/./routeFieldColl_shard4_replica2/data/]
   [junit4]   2> 1072975 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard3_replica1' using configuration from collection routeFieldColl
   [junit4]   2> 1072976 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.c.SolrCore [[routeFieldColl_shard3_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node3/routeFieldColl_shard3_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node3/./routeFieldColl_shard3_replica1/data/]
   [junit4]   2> 1072980 WARN  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard2_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1072981 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1072982 WARN  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.s.IndexSchema [routeFieldColl_shard1_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1072982 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard2_replica2' using configuration from collection routeFieldColl
   [junit4]   2> 1072982 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.s.IndexSchema Loaded schema cloud-dynamic/1.6 with uniqueid field id
   [junit4]   2> 1072983 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.c.SolrCore [[routeFieldColl_shard2_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node2/routeFieldColl_shard2_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node2/./routeFieldColl_shard2_replica2/data/]
   [junit4]   2> 1072984 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'routeFieldColl_shard1_replica2' using configuration from collection routeFieldColl
   [junit4]   2> 1072984 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.c.SolrCore [[routeFieldColl_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node1/routeFieldColl_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.CustomCollectionTest_144CBF0BFE38B622-001/tempDir-001/node1/./routeFieldColl_shard1_replica2/data/]
   [junit4]   2> 1073121 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1073122 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1073123 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1073124 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1073130 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1073130 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1073130 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1073130 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1073132 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1073132 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1073133 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1073133 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1073134 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6a7618d5[routeFieldColl_shard4_replica1] main]
   [junit4]   2> 1073144 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6e7fc61f[routeFieldColl_shard1_replica1] main]
   [junit4]   2> 1073148 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1073148 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1073149 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1073149 INFO  (searcherExecutor-4380-thread-1-processing-n:127.0.0.1:59114_solr x:routeFieldColl_shard1_replica1 s:shard1 c:routeFieldColl) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.SolrCore [routeFieldColl_shard1_replica1] Registered new searcher Searcher@6e7fc61f[routeFieldColl_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1073149 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560061109431435264
   [junit4]   2> 1073150 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1073150 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1073152 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1073152 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1073154 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6d850e32[routeFieldColl_shard2_replica1] main]
   [junit4]   2> 1073156 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1073157 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1073157 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1073157 INFO  (searcherExecutor-4381-thread-1-processing-n:127.0.0.1:59115_solr x:routeFieldColl_shard2_replica1 s:shard2 c:routeFieldColl) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.c.SolrCore [routeFieldColl_shard2_replica1] Registered new searcher Searcher@6d850e32[routeFieldColl_shard2_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1073158 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560061109440872448
   [junit4]   2> 1073163 INFO  (qtp1790660990-10327) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9997ms
   [junit4]   2> 1073166 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1073166 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1073167 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1073168 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1073169 INFO  (qtp609904077-10346) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1073170 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1073170 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1073178 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1073178 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1073180 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1073180 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1073181 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@7a07474c[routeFieldColl_shard4_replica2] main]
   [junit4]   2> 1073183 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1073184 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1073184 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1073184 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560061109468135424
   [junit4]   2> 1073184 INFO  (searcherExecutor-4384-thread-1-processing-n:127.0.0.1:59117_solr x:routeFieldColl_shard4_replica2 s:shard4 c:routeFieldColl) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.SolrCore [routeFieldColl_shard4_replica2] Registered new searcher Searcher@7a07474c[routeFieldColl_shard4_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1073184 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1073184 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1073184 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1073185 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1073185 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1073185 INFO  (searcherExecutor-4382-thread-1-processing-n:127.0.0.1:59115_solr x:routeFieldColl_shard4_replica1 s:shard4 c:routeFieldColl) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.c.SolrCore [routeFieldColl_shard4_replica1] Registered new searcher Searcher@6a7618d5[routeFieldColl_shard4_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1073185 INFO  (qtp609904077-10345) [n:127.0.0.1:59115_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560061109469184000
   [junit4]   2> 1073187 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1073187 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1073188 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1950aea3[routeFieldColl_shard3_replica1] main]
   [junit4]   2> 1073190 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1073191 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1073192 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1073192 INFO  (searcherExecutor-4385-thread-1-processing-n:127.0.0.1:59114_solr x:routeFieldColl_shard3_replica1 s:shard3 c:routeFieldColl) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.c.SolrCore [routeFieldColl_shard3_replica1] Registered new searcher Searcher@1950aea3[routeFieldColl_shard3_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1073191 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1073192 INFO  (qtp1790660990-10328) [n:127.0.0.1:59114_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560061109476524032
   [junit4]   2> 1073193 INFO  (qtp620209634-10333) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard1  x:routeFieldColl_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1073193 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@e23f431[routeFieldColl_shard3_replica2] main]
   [junit4]   2> 1073194 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@37558aff[routeFieldColl_shard2_replica2] main]
   [junit4]   2> 1073195 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1073195 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1073195 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1073196 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560061109480718336
   [junit4]   2> 1073196 INFO  (searcherExecutor-4383-thread-1-processing-n:127.0.0.1:59116_solr x:routeFieldColl_shard3_replica2 s:shard3 c:routeFieldColl) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.SolrCore [routeFieldColl_shard3_replica2] Registered new searcher Searcher@e23f431[routeFieldColl_shard3_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1073199 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 1073202 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1073203 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1073203 INFO  (searcherExecutor-4386-thread-1-processing-n:127.0.0.1:59117_solr x:routeFieldColl_shard2_replica2 s:shard2 c:routeFieldColl) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.c.SolrCore [routeFieldColl_shard2_replica2] Registered new searcher Searcher@37558aff[routeFieldColl_shard2_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1073203 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1073203 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1073203 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:59117/solr/routeFieldColl_shard4_replica2/
   [junit4]   2> 1073203 INFO  (qtp1368165724-10355) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard2  x:routeFieldColl_shard2_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560061109488058368
   [junit4]   2> 1073203 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.PeerSync PeerSync: core=routeFieldColl_shard4_replica2 url=http://127.0.0.1:59117/solr START replicas=[http://127.0.0.1:59115/solr/routeFieldColl_shard4_replica1/] nUpdates=100
   [junit4]   2> 1073204 INFO  (qtp1368165724-10353) [n:127.0.0.1:59117_solr c:routeFieldColl s:shard4  x:routeFieldColl_shard4_replica2] o.a.s.u.PeerSync PeerSync: core=routeFieldColl_shard4_replica2 url=http://127.0.0.1:59117/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1073206 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1073206 INFO  (qtp620209634-10335) [n:127.0.0.1:59116_solr c:routeFieldColl s:shard3  x:routeFieldColl_shard3_replica2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1073206 INFO  (qtp6099

[...truncated too long message...]

 2924742 INFO  (jetty-launcher-4150-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5b5f67f0{/solr,null,UNAVAILABLE}
   [junit4]   2> 2926241 WARN  (zkCallback-4173-thread-1-processing-n:127.0.0.1:63878_solr) [n:127.0.0.1:63878_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2926241 WARN  (zkCallback-4175-thread-2-processing-n:127.0.0.1:63872_solr) [n:127.0.0.1:63872_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2926242 INFO  (jetty-launcher-4150-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2926241 WARN  (zkCallback-4172-thread-1-processing-n:127.0.0.1:63873_solr) [n:127.0.0.1:63873_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2926242 INFO  (jetty-launcher-4150-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6f3940a8{/solr,null,UNAVAILABLE}
   [junit4]   2> 2926242 INFO  (jetty-launcher-4150-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2926242 INFO  (jetty-launcher-4150-thread-4) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2926242 INFO  (jetty-launcher-4150-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@77946d06{/solr,null,UNAVAILABLE}
   [junit4]   2> 2926242 INFO  (jetty-launcher-4150-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@127ec34{/solr,null,UNAVAILABLE}
   [junit4]   2> 2926246 INFO  (SUITE-TestSegmentSorting-seed#[144CBF0BFE38B622]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:63870 63870
   [junit4]   2> 2927412 INFO  (Thread-4012) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:63870 63870
   [junit4]   2> 2931765 WARN  (Thread-4012) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	8	/solr/aliases.json
   [junit4]   2> 	8	/solr/clusterprops.json
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	3	/solr/configs/class org.apache.solr.cloud.TestSegmentSorting_configSet
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	8	/solr/clusterstate.json
   [junit4]   2> 	4	/solr/collections/testAtomicUpdateOfSegmentSortField/state.json
   [junit4]   2> 	3	/solr/collections/testSegmentTerminateEarly/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	35	/solr/overseer/collection-queue-work
   [junit4]   2> 	21	/solr/overseer/queue
   [junit4]   2> 	11	/solr/collections
   [junit4]   2> 	8	/solr/live_nodes
   [junit4]   2> 	4	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 2931765 INFO  (SUITE-TestSegmentSorting-seed#[144CBF0BFE38B622]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestSegmentSorting_144CBF0BFE38B622-001
   [junit4]   2> NOTE: test params are: codec=Lucene62, sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=pl, timezone=Pacific/Nauru
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=311185408,total=531103744
   [junit4]   2> NOTE: All tests run in this JVM: [DebugComponentTest, SimpleCollectionCreateDeleteTest, BaseCdcrDistributedZkTest, TestSolrCoreSnapshots, BigEndianAscendingWordSerializerTest, RemoteQueryErrorTest, CoreAdminHandlerTest, SolrInfoMBeanTest, DistribDocExpirationUpdateProcessorTest, TestConfig, TestHighFrequencyDictionaryFactory, SolrCmdDistributorTest, TestRuleBasedAuthorizationPlugin, DistributedSuggestComponentTest, JavabinLoaderTest, TestSuggestSpellingConverter, CdcrRequestHandlerTest, TestSchemalessBufferedUpdates, SolrCoreCheckLockOnStartupTest, ZkSolrClientTest, TestUtils, TestExportWriter, OpenCloseCoreStressTest, WordBreakSolrSpellCheckerTest, BlockJoinFacetSimpleTest, CacheHeaderTest, TestSolrQueryParser, ReturnFieldsTest, TestRandomFaceting, DeleteStatusTest, MigrateRouteKeyTest, EnumFieldTest, DistributedFacetPivotLongTailTest, DocValuesNotIndexedTest, TestFieldSortValues, TestSQLHandler, TestFileDictionaryLookup, TestInitParams, HdfsRecoveryZkTest, MinimalSchemaTest, TestRandomFlRTGCloud, BlobRepositoryCloudTest, CircularListTest, CloneFieldUpdateProcessorFactoryTest, TestFieldCacheSort, SharedFSAutoReplicaFailoverTest, FieldMutatingUpdateProcessorTest, DeleteLastCustomShardedReplicaTest, TestStressUserVersions, TestPHPSerializedResponseWriter, MetricsHandlerTest, BlockDirectoryTest, TestUniqueKeyFieldResource, TestShortCircuitedRequests, TestManagedSynonymFilterFactory, TestSystemIdResolver, CollectionStateFormat2Test, TestSolrCLIRunExample, CurrencyFieldOpenExchangeTest, TestMiniSolrCloudClusterSSL, NoCacheHeaderTest, TestIndexSearcher, SuggestComponentContextFilterQueryTest, TestRebalanceLeaders, TolerantUpdateProcessorTest, TestSolr4Spatial, PreAnalyzedUpdateProcessorTest, TestFieldCacheVsDocValues, IndexSchemaTest, TestDistributedGrouping, ExternalFileFieldSortTest, TestFieldTypeResource, TestSolrConfigHandler, QueryParsingTest, TestJmxMonitoredMap, RequestHandlersTest, ResourceLoaderTest, BooleanFieldTest, TestMinMaxOnMultiValuedField, DistanceFunctionTest, TestCollapseQParserPlugin, SignatureUpdateProcessorFactoryTest, TestNumericTerms64, TestClassNameShortening, StatsComponentTest, TestSolrConfigHandlerCloud, SuggesterWFSTTest, ConvertedLegacyTest, SegmentsInfoRequestHandlerTest, TestBlobHandler, SolrJmxReporterTest, TestMacroExpander, TestQuerySenderListener, TestConfigSetImmutable, TestSolrQueryParserDefaultOperatorResource, TestInPlaceUpdatesStandalone, LukeRequestHandlerTest, TestLeaderElectionZkExpiry, TestExactStatsCache, TestSolrCloudWithDelegationTokens, TestImplicitCoreProperties, TestMiniSolrCloudCluster, TestChildDocTransformer, TestWriterPerf, ZkStateWriterTest, SolrIndexConfigTest, MultiThreadedOCPTest, TestSolrXml, TestOverriddenPrefixQueryForCustomFieldType, TestSearchPerf, TestLocalFSCloudBackupRestore, TestTolerantUpdateProcessorCloud, OverriddenZkACLAndCredentialsProvidersTest, TestLuceneMatchVersion, ReplicationFactorTest, TestFuzzyAnalyzedSuggestions, TestRestManager, TestSSLRandomization, TestDistribIDF, CustomCollectionTest, TestAuthorizationFramework, SpellingQueryConverterTest, TestRealTimeGet, TestDownShardTolerantSearch, TestDefaultSearchFieldResource, TestSubQueryTransformerCrossCore, TestQuerySenderNoQuery, DistributedIntervalFacetingTest, TestAtomicUpdateErrorCases, TestSolrCoreProperties, TestCloudInspectUtil, TestReloadAndDeleteDocs, ShardRoutingCustomTest, DistributedQueueTest, PreAnalyzedFieldManagedSchemaCloudTest, TestClusterStateMutator, TestApiFramework, SpellPossibilityIteratorTest, CoreMergeIndexesAdminHandlerTest, ExplicitHLLTest, TestPivotHelperCode, AutoCommitTest, SearchHandlerTest, DateMathParserTest, HLLSerializationTest, TestDistributedStatsComponentCardinality, DirectUpdateHandlerOptimizeTest, TestHdfsCloudBackupRestore, BasicDistributedZk2Test, TestOmitPositions, TestBadConfig, TermVectorComponentDistributedTest, TestReqParamsAPI, ScriptEngineTest, SolrMetricReporterTest, ConnectionManagerTest, SuggesterTest, HdfsBasicDistributedZkTest, TestStressInPlaceUpdates, TestSystemCollAutoCreate, TestConfigSets, TestRandomCollapseQParserPlugin, CloudExitableDirectoryReaderTest, TestPathTrie, TestManagedResource, IgnoreCommitOptimizeUpdateProcessorFactoryTest, NotRequiredUniqueKeyTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, LeaderElectionIntegrationTest, BasicZkTest, FullSolrCloudDistribCmdsTest, ZkCLITest, TestRecovery, ZkControllerTest, TestStressReorder, TestJoin, TestReload, TestStressVersions, HardAutoCommitTest, DistributedTermsComponentTest, TestRangeQuery, SolrCoreTest, SpellCheckComponentTest, TestGroupingSearch, BadIndexSchemaTest, TestFiltering, BasicFunctionalityTest, HighlighterTest, ShowFileRequestHandlerTest, SolrIndexSplitterTest, TestCoreDiscovery, SpellCheckCollatorTest, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTSTTest, TestTrie, TestCSVLoader, SchemaVersionSpecificBehaviorTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestSolrDeletionPolicy1, TestSurroundQueryParser, TestQueryUtils, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, TermVectorComponentTest, TestIndexingPerformance, FieldAnalysisRequestHandlerTest, LoggingHandlerTest, TestCollationField, TestCSVResponseWriter, JsonLoaderTest, HighlighterConfigTest, UpdateParamsTest, CopyFieldTest, TestStressRecovery, ConcurrentDeleteAndCreateCollectionTest, DeleteNodeTest, DistribCursorPagingTest, ForceLeaderTest, LeaderFailoverAfterPartitionTest, LeaderInitiatedRecoveryOnShardRestartTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerModifyCollectionTest, OverseerRolesTest, OverseerTaskQueueTest, SSLMigrationTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverUtilsTest, SolrCloudExampleTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TestCloudPivotFacet, TestCloudPseudoReturnFields, TestClusterProperties, TestConfigSetsAPI, TestConfigSetsAPIZkFailure, TestDeleteCollectionOnDownNodes, TestDistribDocBasedVersion, TestExclusionRuleCollectionAccess, TestLockTree, TestRandomRequestDistribution, TestRequestForwarding, TestSegmentSorting]
   [junit4] Completed [520/695 (5!)] on J1 in 50.41s, 2 tests, 1 failure, 1 error <<< FAILURES!

[...truncated 63252 lines...]