You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2020/03/16 10:58:52 UTC

[JENKINS] Lucene-Solr-8.x-MacOSX (64bit/jdk-11.0.6) - Build # 749 - Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/749/
Java: 64bit/jdk-11.0.6 -XX:-UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.core.TestDynamicLoading.testDynamicLoading

Error Message:
{   "responseHeader":{     "status":500,     "QTime":30232},   "errorMessages":["3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]\n"],   "WARNING":"This response format is experimental.  It is likely to change in the future.",   "error":{     "metadata":[       "error-class","org.apache.solr.common.SolrException",       "root-error-class","org.apache.solr.common.SolrException"],     "msg":"3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]",     "trace":"org.apache.solr.common.SolrException: 3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:813)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:104)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:522)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:363)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:161)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:139)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:802)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:579)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:500)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)\n\tat org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n",     "code":500}}  expected null, but was:<[3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/] ]>

Stack Trace:
java.lang.AssertionError: {
  "responseHeader":{
    "status":500,
    "QTime":30232},
  "errorMessages":["3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]\n"],
  "WARNING":"This response format is experimental.  It is likely to change in the future.",
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]",
    "trace":"org.apache.solr.common.SolrException: 3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:813)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:104)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:522)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:363)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:161)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:139)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:802)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:579)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:500)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)\n\tat org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n",
    "code":500}}
 expected null, but was:<[3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]
]>
	at __randomizedtesting.SeedInfo.seed([9F7A4C64D1B825AF:473761332665800F]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotNull(Assert.java:755)
	at org.junit.Assert.assertNull(Assert.java:737)
	at org.apache.solr.core.TestSolrConfigHandler.runConfigCommand(TestSolrConfigHandler.java:179)
	at org.apache.solr.core.TestDynamicLoading.testDynamicLoading(TestDynamicLoading.java:79)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  org.apache.solr.handler.TestSolrConfigHandlerCloud.test

Error Message:
{   "responseHeader":{     "status":500,     "QTime":30186},   "errorMessages":["4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:54229/ck_wwl/nd/collection1_shard2_replica_n1/, http://127.0.0.1:54237/ck_wwl/nd/collection1_shard2_replica_n6/, http://127.0.0.1:54233/ck_wwl/nd/collection1_shard1_replica_n2/, http://127.0.0.1:54241/ck_wwl/nd/collection1_shard1_replica_n5/]\n"],   "WARNING":"This response format is experimental.  It is likely to change in the future.",   "error":{     "metadata":[       "error-class","org.apache.solr.common.SolrException",       "root-error-class","org.apache.solr.common.SolrException"],     "msg":"4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:54229/ck_wwl/nd/collection1_shard2_replica_n1/, http://127.0.0.1:54237/ck_wwl/nd/collection1_shard2_replica_n6/, http://127.0.0.1:54233/ck_wwl/nd/collection1_shard1_replica_n2/, http://127.0.0.1:54241/ck_wwl/nd/collection1_shard1_replica_n5/]",     "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:54229/ck_wwl/nd/collection1_shard2_replica_n1/, http://127.0.0.1:54237/ck_wwl/nd/collection1_shard2_replica_n6/, http://127.0.0.1:54233/ck_wwl/nd/collection1_shard1_replica_n2/, http://127.0.0.1:54241/ck_wwl/nd/collection1_shard1_replica_n5/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:813)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:104)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:522)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:363)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:161)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:139)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:802)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:579)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:500)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n",     "code":500}}  expected null, but was:<[4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:54229/ck_wwl/nd/collection1_shard2_replica_n1/, http://127.0.0.1:54237/ck_wwl/nd/collection1_shard2_replica_n6/, http://127.0.0.1:54233/ck_wwl/nd/collection1_shard1_replica_n2/, http://127.0.0.1:54241/ck_wwl/nd/collection1_shard1_replica_n5/] ]>

Stack Trace:
java.lang.AssertionError: {
  "responseHeader":{
    "status":500,
    "QTime":30186},
  "errorMessages":["4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:54229/ck_wwl/nd/collection1_shard2_replica_n1/, http://127.0.0.1:54237/ck_wwl/nd/collection1_shard2_replica_n6/, http://127.0.0.1:54233/ck_wwl/nd/collection1_shard1_replica_n2/, http://127.0.0.1:54241/ck_wwl/nd/collection1_shard1_replica_n5/]\n"],
  "WARNING":"This response format is experimental.  It is likely to change in the future.",
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:54229/ck_wwl/nd/collection1_shard2_replica_n1/, http://127.0.0.1:54237/ck_wwl/nd/collection1_shard2_replica_n6/, http://127.0.0.1:54233/ck_wwl/nd/collection1_shard1_replica_n2/, http://127.0.0.1:54241/ck_wwl/nd/collection1_shard1_replica_n5/]",
    "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:54229/ck_wwl/nd/collection1_shard2_replica_n1/, http://127.0.0.1:54237/ck_wwl/nd/collection1_shard2_replica_n6/, http://127.0.0.1:54233/ck_wwl/nd/collection1_shard1_replica_n2/, http://127.0.0.1:54241/ck_wwl/nd/collection1_shard1_replica_n5/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:813)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:104)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:522)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:363)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:161)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:139)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:802)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:579)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:500)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n",
    "code":500}}
 expected null, but was:<[4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [http://127.0.0.1:54229/ck_wwl/nd/collection1_shard2_replica_n1/, http://127.0.0.1:54237/ck_wwl/nd/collection1_shard2_replica_n6/, http://127.0.0.1:54233/ck_wwl/nd/collection1_shard1_replica_n2/, http://127.0.0.1:54241/ck_wwl/nd/collection1_shard1_replica_n5/]
]>
	at __randomizedtesting.SeedInfo.seed([9F7A4C64D1B825AF:172E73BE7F444857]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotNull(Assert.java:755)
	at org.junit.Assert.assertNull(Assert.java:737)
	at org.apache.solr.core.TestSolrConfigHandler.runConfigCommand(TestSolrConfigHandler.java:179)
	at org.apache.solr.core.TestSolrConfigHandler.reqhandlertests(TestSolrConfigHandler.java:202)
	at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqHandlerAPIs(TestSolrConfigHandlerCloud.java:80)
	at org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:50)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:834)




Build Log:
[...truncated 13384 lines...]
   [junit4] Suite: org.apache.solr.handler.TestSolrConfigHandlerCloud
   [junit4]   2> 57633 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9F7A4C64D1B825AF]-worker) [     ] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/server/solr/configsets/_default/conf'
   [junit4]   2> 57634 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9F7A4C64D1B825AF]-worker) [     ] o.a.s.SolrTestCaseJ4 Created dataDir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/data-dir-5-001
   [junit4]   2> 57634 WARN  (SUITE-TestSolrConfigHandlerCloud-seed#[9F7A4C64D1B825AF]-worker) [     ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=17 numCloses=17
   [junit4]   2> 57635 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9F7A4C64D1B825AF]-worker) [     ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 57635 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9F7A4C64D1B825AF]-worker) [     ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 57635 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9F7A4C64D1B825AF]-worker) [     ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 57635 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9F7A4C64D1B825AF]-worker) [     ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /ck_wwl/nd
   [junit4]   2> 57678 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 57687 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 57687 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 57802 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer start zk server on port:54168
   [junit4]   2> 57802 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer waitForServerUp: 127.0.0.1:54168
   [junit4]   2> 57806 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:54168
   [junit4]   2> 57807 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 54168
   [junit4]   2> 57918 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 57945 INFO  (zkConnectionManagerCallback-66-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 57945 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 57969 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 57974 INFO  (zkConnectionManagerCallback-68-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 57974 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 58000 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 58016 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 58022 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 58049 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 58077 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 58095 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 58129 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 58148 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 58168 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 58209 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 58259 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkTestServer put /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 58298 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 59214 WARN  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 59215 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 59215 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 59215 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.6+10
   [junit4]   2> 59249 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 59249 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 59249 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 59258 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@13f60fca{/ck_wwl/nd,null,AVAILABLE}
   [junit4]   2> 59264 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.s.AbstractConnector Started ServerConnector@35ea8c0a{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:54179}
   [junit4]   2> 59265 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.s.Server Started @59345ms
   [junit4]   2> 59265 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/ck_wwl/nd, solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/tempDir-001/control/data, hostPort=54179, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/control-001/cores, replicaType=NRT}
   [junit4]   2> 59267 ERROR (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 59267 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 59267 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.6.0
   [junit4]   2> 59267 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 59267 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr
   [junit4]   2> 59267 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-03-16T09:59:02.355268Z
   [junit4]   2> 59270 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 59276 INFO  (zkConnectionManagerCallback-70-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 59278 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 59393 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 59395 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/control-001/solr.xml
   [junit4]   2> 59405 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 59405 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 59411 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 59487 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 59492 WARN  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@4b6a91c3[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 59493 WARN  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@4b6a91c3[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 59515 WARN  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@71eddd98[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 59515 WARN  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@71eddd98[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 59517 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54168/solr
   [junit4]   2> 59568 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 59571 INFO  (zkConnectionManagerCallback-77-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 59571 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 59716 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 59722 INFO  (zkConnectionManagerCallback-79-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 59725 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 60802 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54179_ck_wwl%2Fnd
   [junit4]   2> 60817 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.Overseer Overseer (id=72145233585307652-127.0.0.1:54179_ck_wwl%2Fnd-n_0000000000) starting
   [junit4]   2> 61275 INFO  (OverseerStateUpdate-72145233585307652-127.0.0.1:54179_ck_wwl%2Fnd-n_0000000000) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:54179_ck_wwl%2Fnd
   [junit4]   2> 61284 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54179_ck_wwl%2Fnd
   [junit4]   2> 61303 INFO  (zkCallback-78-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 61395 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 61431 WARN  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 61490 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 61587 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 61600 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 61602 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 61622 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/control-001/cores
   [junit4]   2> 61758 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 61766 INFO  (zkConnectionManagerCallback-88-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61767 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 61781 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 61813 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54168/solr ready
   [junit4]   2> 61819 INFO  (qtp1269418923-306) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:54179_ck_wwl%252Fnd&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 61890 INFO  (OverseerThreadFactory-121-thread-1-processing-n:127.0.0.1:54179_ck_wwl%2Fnd) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 62174 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd    x:control_collection_shard1_replica_n1 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 62175 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd    x:control_collection_shard1_replica_n1 ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 63237 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.6.0
   [junit4]   2> 63283 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.s.IndexSchema Schema name=test
   [junit4]   2> 63488 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 63780 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from configset conf1, trusted=true
   [junit4]   2> 63791 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 63792 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 63797 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=109.7744140625, floorSegmentMB=2.08984375, forceMergeDeletesPctAllowed=2.9431984678398115, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=47.56760251072808
   [junit4]   2> 63817 WARN  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 64053 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 64053 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 64061 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 64061 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 64067 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=45, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 64076 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2e1099cc[control_collection_shard1_replica_n1] main]
   [junit4]   2> 64105 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 64108 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 64109 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 64112 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1661314266247987200
   [junit4]   2> 64132 INFO  (searcherExecutor-126-thread-1-processing-n:127.0.0.1:54179_ck_wwl%2Fnd x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@2e1099cc[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 64228 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 64238 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 64328 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 64329 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 64329 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54179/ck_wwl/nd/control_collection_shard1_replica_n1/
   [junit4]   2> 64355 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 64362 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SyncStrategy http://127.0.0.1:54179/ck_wwl/nd/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 64364 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/72145233585307652-core_node2-n_0000000000
   [junit4]   2> 64417 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54179/ck_wwl/nd/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 64468 INFO  (zkCallback-78-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 64468 INFO  (zkCallback-78-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 64533 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 64558 INFO  (qtp1269418923-308) [n:127.0.0.1:54179_ck_wwl%2Fnd c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2384
   [junit4]   2> 64599 INFO  (qtp1269418923-306) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 64660 INFO  (zkCallback-78-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 64660 INFO  (zkCallback-78-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 64661 INFO  (zkCallback-78-thread-3) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 64682 INFO  (qtp1269418923-306) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:54179_ck_wwl%252Fnd&wt=javabin&version=2} status=0 QTime=2863
   [junit4]   2> 64683 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.AbstractFullDistribZkTestBase Waiting to see 1 active replicas in collection: control_collection
   [junit4]   2> 64846 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 64859 INFO  (zkConnectionManagerCallback-94-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 64859 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 64870 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 64900 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54168/solr ready
   [junit4]   2> 64903 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 64915 INFO  (qtp1269418923-306) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 64975 INFO  (OverseerThreadFactory-121-thread-2-processing-n:127.0.0.1:54179_ck_wwl%2Fnd) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 64981 INFO  (OverseerCollectionConfigSetProcessor-72145233585307652-127.0.0.1:54179_ck_wwl%2Fnd-n_0000000000) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 65263 WARN  (OverseerThreadFactory-121-thread-2-processing-n:127.0.0.1:54179_ck_wwl%2Fnd) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 65275 INFO  (qtp1269418923-306) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 65299 INFO  (qtp1269418923-306) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=384
   [junit4]   2> 65309 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrCloudTestCase active slice count: 2 expected:2
   [junit4]   2> 65309 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrCloudTestCase active replica count: 0 expected replica count: 0
   [junit4]   2> 65309 INFO  (watches-91-thread-1) [     ] o.a.s.c.SolrCloudTestCase active slice count: 2 expected:2
   [junit4]   2> 65309 INFO  (watches-91-thread-1) [     ] o.a.s.c.SolrCloudTestCase active replica count: 0 expected replica count: 0
   [junit4]   2> 65313 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrCloudTestCase active slice count: 2 expected:2
   [junit4]   2> 65313 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrCloudTestCase active replica count: 0 expected replica count: 0
   [junit4]   2> 65316 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrCloudTestCase active slice count: 2 expected:2
   [junit4]   2> 65316 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.SolrCloudTestCase active replica count: 0 expected replica count: 0
   [junit4]   2> 65316 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=4
   [junit4]   2> 66994 INFO  (OverseerCollectionConfigSetProcessor-72145233585307652-127.0.0.1:54179_ck_wwl%2Fnd-n_0000000000) [n:127.0.0.1:54179_ck_wwl%2Fnd     ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 67360 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-1-001 of type NRT for shard2
   [junit4]   2> 67391 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 67391 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 67391 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 67391 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.6+10
   [junit4]   2> 67404 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 67404 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 67404 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 67408 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@740f5fdb{/ck_wwl/nd,null,AVAILABLE}
   [junit4]   2> 67444 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.AbstractConnector Started ServerConnector@1e89b21b{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:54229}
   [junit4]   2> 67444 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.Server Started @67524ms
   [junit4]   2> 67444 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/ck_wwl/nd, solrconfig=solrconfig.xml, solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/tempDir-001/jetty1, hostPort=54229, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-1-001/cores, replicaType=NRT}
   [junit4]   2> 67445 ERROR (closeThreadPool-95-thread-1) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 67445 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 67445 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.6.0
   [junit4]   2> 67445 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 67445 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr
   [junit4]   2> 67445 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-03-16T09:59:10.533358Z
   [junit4]   2> 67457 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 67471 INFO  (zkConnectionManagerCallback-97-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 67471 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 67587 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 67587 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-1-001/solr.xml
   [junit4]   2> 67658 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 67658 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 67672 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 68173 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 68187 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@73cd0af8[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 68187 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@73cd0af8[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 68212 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@3aa5ad1f[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 68213 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@3aa5ad1f[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 68216 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54168/solr
   [junit4]   2> 68240 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 68245 INFO  (zkConnectionManagerCallback-104-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 68248 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 68377 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 68391 INFO  (zkConnectionManagerCallback-106-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 68396 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 68482 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 68620 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.c.ZkController Publish node=127.0.0.1:54229_ck_wwl%2Fnd as DOWN
   [junit4]   2> 68656 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 68657 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54229_ck_wwl%2Fnd
   [junit4]   2> 68673 INFO  (zkCallback-78-thread-3) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 68685 INFO  (zkCallback-93-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 68688 INFO  (zkCallback-105-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 68774 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 68775 WARN  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 68862 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 68993 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 69022 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 69022 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 69024 INFO  (closeThreadPool-95-thread-1) [n:127.0.0.1:54229_ck_wwl%2Fnd     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-1-001/cores
   [junit4]   2> 69192 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.AbstractFullDistribZkTestBase waitForLiveNode: 127.0.0.1:54229_ck_wwl%2Fnd
   [junit4]   2> 69473 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-2-001 of type NRT for shard1
   [junit4]   2> 69486 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 69486 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 69487 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 69487 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.6+10
   [junit4]   2> 69525 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 69526 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 69526 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 69529 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3459610a{/ck_wwl/nd,null,AVAILABLE}
   [junit4]   2> 69556 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.AbstractConnector Started ServerConnector@181c35b9{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:54233}
   [junit4]   2> 69556 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.Server Started @69636ms
   [junit4]   2> 69556 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/ck_wwl/nd, solrconfig=solrconfig.xml, solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/tempDir-001/jetty2, hostPort=54233, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-2-001/cores}
   [junit4]   2> 69556 ERROR (closeThreadPool-95-thread-1) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 69556 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 69557 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.6.0
   [junit4]   2> 69557 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 69557 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr
   [junit4]   2> 69557 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-03-16T09:59:12.645096Z
   [junit4]   2> 69563 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 69575 INFO  (zkConnectionManagerCallback-112-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 69576 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 69727 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 69727 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-2-001/solr.xml
   [junit4]   2> 69777 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 69777 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 69779 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 70019 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 70029 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@27299fe0[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 70029 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@27299fe0[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 70058 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@446776a6[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 70059 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@446776a6[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 70060 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54168/solr
   [junit4]   2> 70068 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 70078 INFO  (zkConnectionManagerCallback-119-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 70079 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 70229 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 70233 INFO  (zkConnectionManagerCallback-121-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 70234 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 70343 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 70509 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:54233_ck_wwl%2Fnd as DOWN
   [junit4]   2> 70564 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 70564 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54233_ck_wwl%2Fnd
   [junit4]   2> 70579 INFO  (zkCallback-78-thread-3) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 70579 INFO  (zkCallback-93-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 70580 INFO  (zkCallback-105-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 70584 INFO  (zkCallback-120-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 70616 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 70617 WARN  (closeThreadPool-95-thread-1) [     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 70750 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 70825 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 70854 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 70861 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde
   [junit4]   2> 70863 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-2-001/cores
   [junit4]   2> 71035 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.AbstractFullDistribZkTestBase waitForLiveNode: 127.0.0.1:54233_ck_wwl%2Fnd
   [junit4]   2> 71232 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9F7A4C64D1B825AF]) [     ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-3-001 of type NRT for shard2
   [junit4]   2> 71239 WARN  (closeThreadPool-95-thread-1) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 71239 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 71239 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 71240 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.6+10
   [junit4]   2> 71256 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 71256 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 71257 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 71261 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4807a33f{/ck_wwl/nd,null,AVAILABLE}
   [junit4]   2> 71317 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.AbstractConnector Started ServerConnector@85279aa{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:54237}
   [junit4]   2> 71317 INFO  (closeThreadPool-95-thread-1) [     ] o.e.j.s.Server Started @71397ms
   [junit4]   2> 71317 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/ck_wwl/nd, solrconfig=solrconfig.xml, solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/tempDir-001/jetty3, hostPort=54237, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-3-001/cores, replicaType=NRT}
   [junit4]   2> 71318 ERROR (closeThreadPool-95-thread-1) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 71318 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 71318 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.6.0
   [junit4]   2> 71318 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 71318 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr
   [junit4]   2> 71318 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-03-16T09:59:14.406311Z
   [junit4]   2> 71324 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 71332 INFO  (zkConnectionManagerCallback-127-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 71334 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 71454 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 71454 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J2/temp/solr.handler.TestSolrConfigHandlerCloud_9F7A4C64D1B825AF-001/shard-3-001/solr.xml
   [junit4]   2> 71476 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 71476 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 71480 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@7ade4dde, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 72168 INFO  (closeThreadPool-95-thread-1) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheck

[...truncated too long message...]

ng -Dtests.method=testDynamicLoading -Dtests.seed=9F7A4C64D1B825AF -Dtests.slow=true -Dtests.locale=en-AG -Dtests.timezone=Europe/Malta -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 78.8s J1 | TestDynamicLoading.testDynamicLoading <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: {
   [junit4]    >   "responseHeader":{
   [junit4]    >     "status":500,
   [junit4]    >     "QTime":30232},
   [junit4]    >   "errorMessages":["3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]\n"],
   [junit4]    >   "WARNING":"This response format is experimental.  It is likely to change in the future.",
   [junit4]    >   "error":{
   [junit4]    >     "metadata":[
   [junit4]    >       "error-class","org.apache.solr.common.SolrException",
   [junit4]    >       "root-error-class","org.apache.solr.common.SolrException"],
   [junit4]    >     "msg":"3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]",
   [junit4]    >     "trace":"org.apache.solr.common.SolrException: 3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:813)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:104)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:522)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:363)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:161)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:139)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:802)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:579)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:500)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)\n\tat org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n",
   [junit4]    >     "code":500}}
   [junit4]    >  expected null, but was:<[3 out of 5 the property overlay to be of version 1 within 30 seconds! Failed cores: [https://127.0.0.1:58374/collection1_shard2_replica_n1/, https://127.0.0.1:58378/collection1_shard1_replica_n2/, https://127.0.0.1:58390/collection1_shard1_replica_n6/]
   [junit4]    > ]>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9F7A4C64D1B825AF:473761332665800F]:0)
   [junit4]    > 	at org.apache.solr.core.TestSolrConfigHandler.runConfigCommand(TestSolrConfigHandler.java:179)
   [junit4]    > 	at org.apache.solr.core.TestDynamicLoading.testDynamicLoading(TestDynamicLoading.java:79)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading_9F7A4C64D1B825AF-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene84): {}, docValues:{}, maxPointsInLeafNode=421, maxMBSortInHeap=7.2065220927051366, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@674fd5d6), locale=en-AG, timezone=Europe/Malta
   [junit4]   2> NOTE: Mac OS X 10.14.6 x86_64/AdoptOpenJDK 11.0.6 (64-bit)/cpus=6,threads=1,free=286434840,total=471859200
   [junit4]   2> NOTE: All tests run in this JVM: [BadIndexSchemaTest, DistributedFacetPivotLargeTest, TestSchemaNameResource, TestSolrConfigHandler, DefaultValueUpdateProcessorTest, TestSQLHandlerNonCloud, VersionInfoTest, UninvertDocValuesMergePolicyTest, SolrTestCaseJ4Test, TestHighFrequencyDictionaryFactory, BasicDistributedZk2Test, QueryParsingTest, TestCoreAdminApis, MultiSolrCloudTestCaseTest, TestNumericRangeQuery64, TestCoreContainer, RoutingToNodesWithPropertiesTest, OverriddenZkACLAndCredentialsProvidersTest, AutoscalingHistoryHandlerTest, TestHttpShardHandlerFactory, ConnectionReuseTest, TestUseDocValuesAsStored, TestSimScenario, TestWithCollection, CdcrVersionReplicationTest, TestDynamicLoading]
   [junit4] Completed [137/901 (2!)] on J1 in 78.85s, 1 test, 1 failure <<< FAILURES!

[...truncated 50596 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/749/consoleText

[repro] Revision: 1d7c6be57223c394b061c228f4803b297b7181b0

[repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseParallelGC"
[repro] Repro line:  ant test  -Dtestcase=TestSolrConfigHandlerCloud -Dtests.method=test -Dtests.seed=9F7A4C64D1B825AF -Dtests.slow=true -Dtests.locale=vi-VN -Dtests.timezone=America/Catamarca -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=TestDynamicLoading -Dtests.method=testDynamicLoading -Dtests.seed=9F7A4C64D1B825AF -Dtests.slow=true -Dtests.locale=en-AG -Dtests.timezone=Europe/Malta -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] JUnit rest result XML files will be moved to: ./repro-reports
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TestDynamicLoading
[repro]       TestSolrConfigHandlerCloud
[repro] ant compile-test

[...truncated 3448 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 -Dtests.class="*.TestDynamicLoading|*.TestSolrConfigHandlerCloud" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseParallelGC" -Dtests.seed=9F7A4C64D1B825AF -Dtests.slow=true -Dtests.locale=en-AG -Dtests.timezone=Europe/Malta -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[...truncated 111 lines...]
[repro] Failures w/original seeds:
[repro]   0/5 failed: org.apache.solr.core.TestDynamicLoading
[repro]   0/5 failed: org.apache.solr.handler.TestSolrConfigHandlerCloud
[repro] Exiting with code 0

[...truncated 73 lines...]

[JENKINS] Lucene-Solr-8.x-MacOSX (64bit/jdk1.8.0) - Build # 751 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/751/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore

Error Message:
ObjectTracker found 13 object(s) that were not released!!! [InternalHttpClient, MMapDirectory, InternalHttpClient, InternalHttpClient, SolrCore, MMapDirectory, MMapDirectory, InternalHttpClient, SolrCore, MMapDirectory, MMapDirectory, MMapDirectory, MMapDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)  at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1241)  at org.apache.solr.cloud.ReplicateFromLeader.startReplication(ReplicateFromLeader.java:109)  at org.apache.solr.cloud.ZkController.startReplicationFromLeader(ZkController.java:1320)  at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:694)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:337)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:424)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1210)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)  at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:251)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:393)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:339)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)  at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:251)  at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:663)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:337)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1074)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:98)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:800)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:540)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:977)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)  at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1241)  at org.apache.solr.cloud.ReplicateFromLeader.startReplication(ReplicateFromLeader.java:109)  at org.apache.solr.cloud.ZkController.startReplicationFromLeader(ZkController.java:1320)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:411)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:339)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1074)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:98)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:800)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:392)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:768)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:424)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1210)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:392)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:768)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)   expected null, but was:<ObjectTracker found 13 object(s) that were not released!!! [InternalHttpClient, MMapDirectory, InternalHttpClient, InternalHttpClient, SolrCore, MMapDirectory, MMapDirectory, InternalHttpClient, SolrCore, MMapDirectory, MMapDirectory, MMapDirectory, MMapDirectory] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)  at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1241)  at org.apache.solr.cloud.ReplicateFromLeader.startReplication(ReplicateFromLeader.java:109)  at org.apache.solr.cloud.ZkController.startReplicationFromLeader(ZkController.java:1320)  at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:694)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:337)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:424)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1210)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)  at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:251)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:393)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:339)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)  at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:251)  at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:663)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:337)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1074)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:98)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:800)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:540)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:977)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)  at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)  at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1241)  at org.apache.solr.cloud.ReplicateFromLeader.startReplication(ReplicateFromLeader.java:109)  at org.apache.solr.cloud.ZkController.startReplicationFromLeader(ZkController.java:1320)  at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:411)  at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:339)  at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1074)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:98)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:800)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:392)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:768)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:424)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1210)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:392)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:768)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)  at org.eclipse.jetty.server.Server.handle(Server.java:500)  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)  at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)  at java.lang.Thread.run(Thread.java:748)  >

Stack Trace:
java.lang.AssertionError: ObjectTracker found 13 object(s) that were not released!!! [InternalHttpClient, MMapDirectory, InternalHttpClient, InternalHttpClient, SolrCore, MMapDirectory, MMapDirectory, InternalHttpClient, SolrCore, MMapDirectory, MMapDirectory, MMapDirectory, MMapDirectory]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
	at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)
	at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)
	at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1241)
	at org.apache.solr.cloud.ReplicateFromLeader.startReplication(ReplicateFromLeader.java:109)
	at org.apache.solr.cloud.ZkController.startReplicationFromLeader(ZkController.java:1320)
	at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:694)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:337)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:424)
	at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1210)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
	at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)
	at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
	at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:251)
	at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:393)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:339)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
	at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)
	at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
	at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:251)
	at org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:663)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:337)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1074)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)
	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)
	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
	at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:98)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:800)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)
	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)
	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
	at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:540)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:977)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)
	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)
	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
	at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:230)
	at org.apache.solr.handler.IndexFetcher.<init>(IndexFetcher.java:272)
	at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1241)
	at org.apache.solr.cloud.ReplicateFromLeader.startReplication(ReplicateFromLeader.java:109)
	at org.apache.solr.cloud.ZkController.startReplicationFromLeader(ZkController.java:1320)
	at org.apache.solr.cloud.RecoveryStrategy.doReplicateOnlyRecovery(RecoveryStrategy.java:411)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:339)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:318)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1074)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)
	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)
	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ScopedHandler

[...truncated too long message...]

vletHandler.java:1604)
   [junit4]    > 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
   [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    > 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]    > 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
   [junit4]    > 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
   [junit4]    > 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]    > 	at org.eclipse.jetty.server.Server.handle(Server.java:500)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
   [junit4]    > 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)
   [junit4]    > 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351)
   [junit4]    > 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:424)
   [junit4]    > 	at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1210)
   [junit4]    > 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]    > 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
   [junit4]    > 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
   [junit4]    > 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:392)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:768)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:985)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:904)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1282)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1193)
   [junit4]    > 	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)
   [junit4]    > 	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
   [junit4]    > 	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
   [junit4]    > 	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
   [junit4]    > 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
   [junit4]    > 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
   [junit4]    > 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
   [junit4]    > 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
   [junit4]    > 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
   [junit4]    > 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]    > 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
   [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    > 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]    > 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
   [junit4]    > 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
   [junit4]    > 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]    > 	at org.eclipse.jetty.server.Server.handle(Server.java:500)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > >
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B21E83B95D0950FE]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:333)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [578/901 (1!)] on J2 in 119.17s, 2 tests, 1 failure <<< FAILURES!

[...truncated 53755 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/751/consoleText

[repro] Revision: 33cd6909df28d39eedf2f777082ba26cf20a59ba

[repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC"
[repro] Repro line:  ant test  -Dtestcase=TestHdfsCloudBackupRestore -Dtests.seed=B21E83B95D0950FE -Dtests.slow=true -Dtests.locale=cs-CZ -Dtests.timezone=Pacific/Kwajalein -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] JUnit rest result XML files will be moved to: ./repro-reports
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       TestHdfsCloudBackupRestore
[repro] ant compile-test

[...truncated 3443 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestHdfsCloudBackupRestore" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC" -Dtests.seed=B21E83B95D0950FE -Dtests.slow=true -Dtests.locale=cs-CZ -Dtests.timezone=Pacific/Kwajalein -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[...truncated 86 lines...]
[repro] Failures w/original seeds:
[repro]   0/5 failed: org.apache.solr.cloud.api.collections.TestHdfsCloudBackupRestore
[repro] Exiting with code 0

[...truncated 73 lines...]

[JENKINS] Lucene-Solr-8.x-MacOSX (64bit/jdk-11.0.6) - Build # 750 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/750/
Java: 64bit/jdk-11.0.6 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.ScheduledTriggerIntegrationTest.testScheduledTrigger

Error Message:
ScheduledTrigger did not fire in time

Stack Trace:
java.lang.AssertionError: ScheduledTrigger did not fire in time
	at __randomizedtesting.SeedInfo.seed([3FB101D065519D0B:ACAA49A23BACC63F]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggerIntegrationTest.testScheduledTrigger(ScheduledTriggerIntegrationTest.java:120)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.join.BlockJoinFacetDistribTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest:     1) Thread[id=30736, name=SessionTracker, state=TIMED_WAITING, group=TGRP-BlockJoinFacetDistribTest]         at java.base@11.0.6/java.lang.Thread.sleep(Native Method)         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest: 
   1) Thread[id=30736, name=SessionTracker, state=TIMED_WAITING, group=TGRP-BlockJoinFacetDistribTest]
        at java.base@11.0.6/java.lang.Thread.sleep(Native Method)
        at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)
	at __randomizedtesting.SeedInfo.seed([3FB101D065519D0B]:0)




Build Log:
[...truncated 15314 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.ScheduledTriggerIntegrationTest
   [junit4]   2> 2117067 INFO  (SUITE-ScheduledTriggerIntegrationTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/server/solr/configsets/_default/conf'
   [junit4]   2> 2117068 INFO  (SUITE-ScheduledTriggerIntegrationTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.SolrTestCaseJ4 Created dataDir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/data-dir-108-001
   [junit4]   2> 2117068 WARN  (SUITE-ScheduledTriggerIntegrationTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=14 numCloses=14
   [junit4]   2> 2117068 INFO  (SUITE-ScheduledTriggerIntegrationTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2117069 INFO  (SUITE-ScheduledTriggerIntegrationTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 2117069 INFO  (SUITE-ScheduledTriggerIntegrationTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2117071 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testScheduledTrigger
   [junit4]   2> 2117071 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/tempDir-001
   [junit4]   2> 2117071 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2117072 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2117072 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2117172 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.ZkTestServer start zk server on port:65345
   [junit4]   2> 2117172 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.ZkTestServer waitForServerUp: 127.0.0.1:65345
   [junit4]   2> 2117172 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:65345
   [junit4]   2> 2117172 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 65345
   [junit4]   2> 2117174 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117178 INFO  (zkConnectionManagerCallback-8777-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117178 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117183 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117187 INFO  (zkConnectionManagerCallback-8779-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117188 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117189 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117192 INFO  (zkConnectionManagerCallback-8781-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117192 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117308 WARN  (jetty-launcher-8782-thread-2) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 2117308 WARN  (jetty-launcher-8782-thread-1) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 2117308 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 2117308 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2117308 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 2117308 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2117308 INFO  (jetty-launcher-8782-thread-2) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.6+10
   [junit4]   2> 2117308 INFO  (jetty-launcher-8782-thread-1) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.6+10
   [junit4]   2> 2117310 INFO  (jetty-launcher-8782-thread-2) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2117310 INFO  (jetty-launcher-8782-thread-2) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2117310 INFO  (jetty-launcher-8782-thread-2) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2117310 INFO  (jetty-launcher-8782-thread-1) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2117310 INFO  (jetty-launcher-8782-thread-1) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2117310 INFO  (jetty-launcher-8782-thread-1) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 2117310 INFO  (jetty-launcher-8782-thread-2) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@23672b9b{/solr,null,AVAILABLE}
   [junit4]   2> 2117310 INFO  (jetty-launcher-8782-thread-1) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@61831e82{/solr,null,AVAILABLE}
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-2) [     ] o.e.j.s.AbstractConnector Started ServerConnector@73eeeecd{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:65352}
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-2) [     ] o.e.j.s.Server Started @1971542ms
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-1) [     ] o.e.j.s.AbstractConnector Started ServerConnector@4e99b554{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:65351}
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-1) [     ] o.e.j.s.Server Started @1971542ms
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65352}
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65351}
   [junit4]   2> 2117311 ERROR (jetty-launcher-8782-thread-2) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2117311 ERROR (jetty-launcher-8782-thread-1) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.6.0
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.6.0
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-03-16T17:04:54.539631Z
   [junit4]   2> 2117311 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-03-16T17:04:54.539630Z
   [junit4]   2> 2117312 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117312 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117314 INFO  (zkConnectionManagerCallback-8784-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117314 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117314 INFO  (zkConnectionManagerCallback-8786-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117314 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117315 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2117315 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2117491 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2117491 WARN  (jetty-launcher-8782-thread-2) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@6e7d8c70[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2117491 WARN  (jetty-launcher-8782-thread-2) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@6e7d8c70[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2117494 WARN  (jetty-launcher-8782-thread-2) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@731e8067[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2117494 WARN  (jetty-launcher-8782-thread-2) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@731e8067[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2117495 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65345/solr
   [junit4]   2> 2117495 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117498 INFO  (zkConnectionManagerCallback-8793-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117498 INFO  (jetty-launcher-8782-thread-2) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117622 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2117623 WARN  (jetty-launcher-8782-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@5897f56b[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2117623 WARN  (jetty-launcher-8782-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@5897f56b[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2117624 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117627 WARN  (jetty-launcher-8782-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@90d6100[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2117627 WARN  (jetty-launcher-8782-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@90d6100[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2117627 INFO  (zkConnectionManagerCallback-8797-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117627 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117628 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65345/solr
   [junit4]   2> 2117630 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117633 INFO  (zkConnectionManagerCallback-8802-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117633 INFO  (jetty-launcher-8782-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117738 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117743 INFO  (zkConnectionManagerCallback-8806-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117743 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117780 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65351_solr
   [junit4]   2> 2117782 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.c.Overseer Overseer (id=72146898448154632-127.0.0.1:65351_solr-n_0000000000) starting
   [junit4]   2> 2117802 INFO  (OverseerStateUpdate-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [n:127.0.0.1:65351_solr     ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:65351_solr
   [junit4]   2> 2117802 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65351_solr
   [junit4]   2> 2117802 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
   [junit4]   2> 2117805 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 2117805 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
   [junit4]   2> 2117805 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 2117806 INFO  (OverseerStateUpdate-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2117810 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 2117811 WARN  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 2117823 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:65351_solr]
   [junit4]   2> 2117825 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers
   [junit4]   2> 2117827 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 2117827 DEBUG (ScheduledTrigger-6425-thread-1) [     ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
   [junit4]   2> 2117829 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2117850 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65351.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cbd48a0
   [junit4]   2> 2117852 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2117858 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65351.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cbd48a0
   [junit4]   2> 2117858 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65351.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cbd48a0
   [junit4]   2> 2117859 INFO  (jetty-launcher-8782-thread-1) [n:127.0.0.1:65351_solr     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/tempDir-001/node1/.
   [junit4]   2> 2117862 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.c.ZkController Publish node=127.0.0.1:65352_solr as DOWN
   [junit4]   2> 2117865 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2117865 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65352_solr
   [junit4]   2> 2117867 INFO  (zkCallback-8805-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2117867 INFO  (zkCallback-8796-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2117871 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 2117872 WARN  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 2117889 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 2117906 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65352.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cbd48a0
   [junit4]   2> 2117914 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65352.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cbd48a0
   [junit4]   2> 2117914 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65352.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cbd48a0
   [junit4]   2> 2117915 INFO  (jetty-launcher-8782-thread-2) [n:127.0.0.1:65352_solr     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/tempDir-001/node2/.
   [junit4]   2> 2117934 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.MiniSolrCloudCluster waitForAllNodes: numServers=2
   [junit4]   2> 2117935 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2117938 INFO  (zkConnectionManagerCallback-8817-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2117938 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2117942 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2117944 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:65345/solr ready
   [junit4]   2> 2117953 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2117958 INFO  (qtp1079919407-25655) [n:127.0.0.1:65351_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2117959 INFO  (qtp2099842840-25652) [n:127.0.0.1:65352_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2117959 DEBUG (qtp1079919407-25651) [n:127.0.0.1:65351_solr     ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 2117960 INFO  (qtp1079919407-25651) [n:127.0.0.1:65351_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 2117960 DEBUG (zkCallback-8805-thread-1) [     ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 2117961 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 2117961 INFO  (qtp2099842840-25648) [n:127.0.0.1:65352_solr     ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&maxShardsPerNode=5&name=testScheduledTrigger&nrtReplicas=3&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2117967 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers
   [junit4]   2> 2117970 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 2117972 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.a.c.CreateCollectionCmd Create collection testScheduledTrigger
   [junit4]   2> 2117975 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/managed-schema to /configs/testScheduledTrigger.AUTOCREATED/managed-schema
   [junit4]   2> 2117980 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/protwords.txt to /configs/testScheduledTrigger.AUTOCREATED/protwords.txt
   [junit4]   2> 2117989 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_it.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/contractions_it.txt
   [junit4]   2> 2117995 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ca.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/contractions_ca.txt
   [junit4]   2> 2117999 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stemdict_nl.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stemdict_nl.txt
   [junit4]   2> 2118003 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_no.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_no.txt
   [junit4]   2> 2118010 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hy.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_hy.txt
   [junit4]   2> 2118016 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_id.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_id.txt
   [junit4]   2> 2118020 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_et.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_et.txt
   [junit4]   2> 2118029 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_da.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_da.txt
   [junit4]   2> 2118034 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hi.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_hi.txt
   [junit4]   2> 2118039 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ga.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_ga.txt
   [junit4]   2> 2118046 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_pt.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_pt.txt
   [junit4]   2> 2118048 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ja.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_ja.txt
   [junit4]   2> 2118055 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hu.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_hu.txt
   [junit4]   2> 2118060 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_el.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_el.txt
   [junit4]   2> 2118065 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ru.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_ru.txt
   [junit4]   2> 2118069 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_tr.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_tr.txt
   [junit4]   2> 2118074 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ar.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_ar.txt
   [junit4]   2> 2118079 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/userdict_ja.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/userdict_ja.txt
   [junit4]   2> 2118083 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_eu.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_eu.txt
   [junit4]   2> 2118087 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_it.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_it.txt
   [junit4]   2> 2118093 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_cz.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_cz.txt
   [junit4]   2> 2118099 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stoptags_ja.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stoptags_ja.txt
   [junit4]   2> 2118102 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_fr.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/contractions_fr.txt
   [junit4]   2> 2118107 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_de.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_de.txt
   [junit4]   2> 2118116 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fa.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_fa.txt
   [junit4]   2> 2118121 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ga.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/contractions_ga.txt
   [junit4]   2> 2118125 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ca.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_ca.txt
   [junit4]   2> 2118130 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_nl.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_nl.txt
   [junit4]   2> 2118135 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_bg.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_bg.txt
   [junit4]   2> 2118138 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_es.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_es.txt
   [junit4]   2> 2118142 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_lv.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_lv.txt
   [junit4]   2> 2118145 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_en.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_en.txt
   [junit4]   2> 2118148 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ro.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_ro.txt
   [junit4]   2> 2118151 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_th.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_th.txt
   [junit4]   2> 2118155 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fr.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_fr.txt
   [junit4]   2> 2118158 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_sv.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_sv.txt
   [junit4]   2> 2118161 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fi.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_fi.txt
   [junit4]   2> 2118164 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/hyphenations_ga.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/hyphenations_ga.txt
   [junit4]   2> 2118168 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_gl.txt to /configs/testScheduledTrigger.AUTOCREATED/lang/stopwords_gl.txt
   [junit4]   2> 2118171 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/solrconfig.xml to /configs/testScheduledTrigger.AUTOCREATED/solrconfig.xml
   [junit4]   2> 2118174 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/synonyms.txt to /configs/testScheduledTrigger.AUTOCREATED/synonyms.txt
   [junit4]   2> 2118176 INFO  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/stopwords.txt to /configs/testScheduledTrigger.AUTOCREATED/stopwords.txt
   [junit4]   2> 2118287 WARN  (OverseerThreadFactory-6427-thread-1-processing-n:127.0.0.1:65351_solr) [n:127.0.0.1:65351_solr     ] o.a.s.c.a.c.CreateCollectionCmd Specified number of replicas of 3 on collection testScheduledTrigger is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 2118295 INFO  (OverseerStateUpdate-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [n:127.0.0.1:65351_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testScheduledTrigger",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testScheduledTrigger_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:65352/solr",
   [junit4]   2>   "node_name":"127.0.0.1:65352_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2118307 INFO  (OverseerStateUpdate-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [n:127.0.0.1:65351_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testScheduledTrigger",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testScheduledTrigger_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:65351/solr",
   [junit4]   2>   "node_name":"127.0.0.1:65351_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2118317 INFO  (OverseerStateUpdate-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [n:127.0.0.1:65351_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testScheduledTrigger",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testScheduledTrigger_shard1_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:65352/solr",
   [junit4]   2>   "node_name":"127.0.0.1:65352_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2118520 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr    x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=testScheduledTrigger.AUTOCREATED&newCollection=true&name=testScheduledTrigger_shard1_replica_n1&action=CREATE&numShards=1&collection=testScheduledTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2118521 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr    x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=testScheduledTrigger.AUTOCREATED&newCollection=true&name=testScheduledTrigger_shard1_replica_n2&action=CREATE&numShards=1&collection=testScheduledTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2118521 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr    x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2118521 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr    x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=testScheduledTrigger.AUTOCREATED&newCollection=true&name=testScheduledTrigger_shard1_replica_n4&action=CREATE&numShards=1&collection=testScheduledTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2118559 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.6.0
   [junit4]   2> 2118559 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.6.0
   [junit4]   2> 2118559 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.6.0
   [junit4]   2> 2118588 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.s.IndexSchema Schema name=default-config
   [junit4]   2> 2118588 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.s.IndexSchema Schema name=default-config
   [junit4]   2> 2118588 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.s.IndexSchema Schema name=default-config
   [junit4]   2> 2118832 DEBUG (ScheduledTrigger-6425-thread-2) [     ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 1
   [junit4]   2> 2118882 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 2118888 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 2118888 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 2118988 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.CoreContainer Creating SolrCore 'testScheduledTrigger_shard1_replica_n2' using configuration from configset testScheduledTrigger.AUTOCREATED, trusted=true
   [junit4]   2> 2118988 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65351.solr.core.testScheduledTrigger.shard1.replica_n2' (registry 'solr.core.testScheduledTrigger.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cbd48a0
   [junit4]   2> 2118989 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.SolrCore [[testScheduledTrigger_shard1_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/tempDir-001/node1/./testScheduledTrigger_shard1_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/tempDir-001/node1/./testScheduledTrigger_shard1_replica_n2/data/]
   [junit4]   2> 2119003 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.CoreContainer Creating SolrCore 'testScheduledTrigger_shard1_replica_n1' using configuration from configset testScheduledTrigger.AUTOCREATED, trusted=true
   [junit4]   2> 2119003 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65352.solr.core.testScheduledTrigger.shard1.replica_n1' (registry 'solr.core.testScheduledTrigger.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cbd48a0
   [junit4]   2> 2119004 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.SolrCore [[testScheduledTrigger_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/tempDir-001/node2/./testScheduledTrigger_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/tempDir-001/node2/./testScheduledTrigger_shard1_replica_n1/data/]
   [junit4]   2> 2119004 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.c.CoreContainer Creating SolrCore 'testScheduledTrigger_shard1_replica_n4' using configuration from configset testScheduledTrigger.AUTOCREATED, trusted=true
   [junit4]   2> 2119005 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65352.solr.core.testScheduledTrigger.shard1.replica_n4' (registry 'solr.core.testScheduledTrigger.shard1.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cbd48a0
   [junit4]   2> 2119005 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.c.SolrCore [[testScheduledTrigger_shard1_replica_n4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/tempDir-001/node2/./testScheduledTrigger_shard1_replica_n4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.ScheduledTriggerIntegrationTest_3FB101D065519D0B-001/tempDir-001/node2/./testScheduledTrigger_shard1_replica_n4/data/]
   [junit4]   2> 2119066 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2119066 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2119067 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2119067 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2119069 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 
   [junit4]   2> 2119069 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2119069 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 
   [junit4]   2> 2119070 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2119070 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6d1aa7cd[testScheduledTrigger_shard1_replica_n4] main]
   [junit4]   2> 2119071 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.s.SolrIndexSearcher Opening [Searcher@36a7b749[testScheduledTrigger_shard1_replica_n1] main]
   [junit4]   2> 2119074 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/testScheduledTrigger.AUTOCREATED
   [junit4]   2> 2119074 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/testScheduledTrigger.AUTOCREATED
   [junit4]   2> 2119075 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/testScheduledTrigger.AUTOCREATED
   [junit4]   2> 2119075 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/testScheduledTrigger.AUTOCREATED/managed-schema
   [junit4]   2> 2119076 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/testScheduledTrigger.AUTOCREATED
   [junit4]   2> 2119076 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/testScheduledTrigger.AUTOCREATED/managed-schema
   [junit4]   2> 2119077 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 2119077 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 2119078 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2119078 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2119078 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 2119078 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 2119079 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2119079 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2119079 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1661341056428408832
   [junit4]   2> 2119079 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1661341056428408832
   [junit4]   2> 2119081 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2119081 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2119083 INFO  (searcherExecutor-6438-thread-1-processing-n:127.0.0.1:65352_solr x:testScheduledTrigger_shard1_replica_n4 c:testScheduledTrigger s:shard1 r:core_node6) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@6d1aa7cd[testScheduledTrigger_shard1_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2119083 INFO  (searcherExecutor-6438-thread-1-processing-n:127.0.0.1:65352_solr x:testScheduledTrigger_shard1_replica_n4 c:testScheduledTrigger s:shard1 r:core_node6) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 2119083 INFO  (searcherExecutor-6438-thread-1-processing-n:127.0.0.1:65352_solr x:testScheduledTrigger_shard1_replica_n4 c:testScheduledTrigger s:shard1 r:core_node6) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 2119083 INFO  (searcherExecutor-6437-thread-1-processing-n:127.0.0.1:65352_solr x:testScheduledTrigger_shard1_replica_n1 c:testScheduledTrigger s:shard1 r:core_node3) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@36a7b749[testScheduledTrigger_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2119083 INFO  (searcherExecutor-6437-thread-1-processing-n:127.0.0.1:65352_solr x:testScheduledTrigger_shard1_replica_n1 c:testScheduledTrigger s:shard1 r:core_node3) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 2119083 INFO  (searcherExecutor-6437-thread-1-processing-n:127.0.0.1:65352_solr x:testScheduledTrigger_shard1_replica_n1 c:testScheduledTrigger s:shard1 r:core_node3) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 2119084 INFO  (searcherExecutor-6437-thread-1-processing-n:127.0.0.1:65352_solr x:testScheduledTrigger_shard1_replica_n1 c:testScheduledTrigger s:shard1 r:core_node3) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.SolrCore [testScheduledTrigger_shard1_replica_n1] Registered new searcher Searcher@36a7b749[testScheduledTrigger_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2119085 INFO  (searcherExecutor-6438-thread-1-processing-n:127.0.0.1:65352_solr x:testScheduledTrigger_shard1_replica_n4 c:testScheduledTrigger s:shard1 r:core_node6) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.c.SolrCore [testScheduledTrigger_shard1_replica_n4] Registered new searcher Searcher@6d1aa7cd[testScheduledTrigger_shard1_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2119085 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 
   [junit4]   2> 2119085 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2119086 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.s.SolrIndexSearcher Opening [Searcher@12856228[testScheduledTrigger_shard1_replica_n2] main]
   [junit4]   2> 2119090 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/testScheduledTrigger.AUTOCREATED
   [junit4]   2> 2119091 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/testScheduledTrigger.AUTOCREATED
   [junit4]   2> 2119091 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/testScheduledTrigger.AUTOCREATED/managed-schema
   [junit4]   2> 2119092 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testScheduledTrigger/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 2119093 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/testScheduledTrigger/leaders/shard1
   [junit4]   2> 2119093 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 2119094 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2119094 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 2119095 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testScheduledTrigger/terms/shard1 to Terms{values={core_node6=0, core_node3=0}, version=1}
   [junit4]   2> 2119095 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/testScheduledTrigger/leaders/shard1
   [junit4]   2> 2119095 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2119096 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1661341056446234624
   [junit4]   2> 2119102 INFO  (searcherExecutor-6436-thread-1-processing-n:127.0.0.1:65351_solr x:testScheduledTrigger_shard1_replica_n2 c:testScheduledTrigger s:shard1 r:core_node5) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@12856228[testScheduledTrigger_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2119102 INFO  (searcherExecutor-6436-thread-1-processing-n:127.0.0.1:65351_solr x:testScheduledTrigger_shard1_replica_n2 c:testScheduledTrigger s:shard1 r:core_node5) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 2119102 INFO  (searcherExecutor-6436-thread-1-processing-n:127.0.0.1:65351_solr x:testScheduledTrigger_shard1_replica_n2 c:testScheduledTrigger s:shard1 r:core_node5) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 2119102 INFO  (searcherExecutor-6436-thread-1-processing-n:127.0.0.1:65351_solr x:testScheduledTrigger_shard1_replica_n2 c:testScheduledTrigger s:shard1 r:core_node5) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.SolrCore [testScheduledTrigger_shard1_replica_n2] Registered new searcher Searcher@12856228[testScheduledTrigger_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2119115 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.ZkShardTerms Successful update of terms at /collections/testScheduledTrigger/terms/shard1 to Terms{values={core_node6=0, core_node3=0, core_node5=0}, version=2}
   [junit4]   2> 2119115 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/testScheduledTrigger/leaders/shard1
   [junit4]   2> 2119115 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=3 found=2 timeoutin=9997ms
   [junit4]   2> 2119620 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2119620 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2119620 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:65352/solr/testScheduledTrigger_shard1_replica_n1/
   [junit4]   2> 2119622 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.u.PeerSync PeerSync: core=testScheduledTrigger_shard1_replica_n1 url=http://127.0.0.1:65352/solr START replicas=[http://127.0.0.1:65351/solr/testScheduledTrigger_shard1_replica_n2/, http://127.0.0.1:65352/solr/testScheduledTrigger_shard1_replica_n4/] nUpdates=100
   [junit4]   2> 2119622 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.u.PeerSync PeerSync: core=testScheduledTrigger_shard1_replica_n1 url=http://127.0.0.1:65352/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2119624 INFO  (qtp2099842840-25652) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.c.S.Request [testScheduledTrigger_shard1_replica_n4]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2119624 INFO  (qtp1079919407-25655) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.c.S.Request [testScheduledTrigger_shard1_replica_n2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 2119625 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 2119625 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2119625 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/testScheduledTrigger/leaders/shard1/leader after winning as /collections/testScheduledTrigger/leader_elect/shard1/election/72146898448154630-core_node3-n_0000000000
   [junit4]   2> 2119629 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:65352/solr/testScheduledTrigger_shard1_replica_n1/ shard1
   [junit4]   2> 2119737 INFO  (zkCallback-8796-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testScheduledTrigger/state.json] for collection [testScheduledTrigger] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2119741 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2119747 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node3 x:testScheduledTrigger_shard1_replica_n1 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=testScheduledTrigger.AUTOCREATED&newCollection=true&name=testScheduledTrigger_shard1_replica_n1&action=CREATE&numShards=1&collection=testScheduledTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1227
   [junit4]   2> 2119846 DEBUG (ScheduledTrigger-6425-thread-2) [     ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2 and last live nodes: 2
   [junit4]   2> 2119847 INFO  (zkCallback-8796-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testScheduledTrigger/state.json] for collection [testScheduledTrigger] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2119847 INFO  (zkCallback-8796-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testScheduledTrigger/state.json] for collection [testScheduledTrigger] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2120120 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr c:testScheduledTrigger s:shard1 r:core_node6 x:testScheduledTrigger_shard1_replica_n4 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=testScheduledTrigger.AUTOCREATED&newCollection=true&name=testScheduledTrigger_shard1_replica_n4&action=CREATE&numShards=1&collection=testScheduledTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1599
   [junit4]   2> 2120127 INFO  (qtp1079919407-25649) [n:127.0.0.1:65351_solr c:testScheduledTrigger s:shard1 r:core_node5 x:testScheduledTrigger_shard1_replica_n2 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=testScheduledTrigger.AUTOCREATED&newCollection=true&name=testScheduledTrigger_shard1_replica_n2&action=CREATE&numShards=1&collection=testScheduledTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1606
   [junit4]   2> 2120130 INFO  (qtp2099842840-25648) [n:127.0.0.1:65352_solr     ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 2120232 INFO  (zkCallback-8796-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testScheduledTrigger/state.json] for collection [testScheduledTrigger] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2120232 INFO  (zkCallback-8805-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testScheduledTrigger/state.json] for collection [testScheduledTrigger] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2120232 INFO  (zkCallback-8805-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testScheduledTrigger/state.json] for collection [testScheduledTrigger] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2120232 INFO  (zkCallback-8796-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testScheduledTrigger/state.json] for collection [testScheduledTrigger] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2120233 INFO  (qtp2099842840-25648) [n:127.0.0.1:65352_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&maxShardsPerNode=5&name=testScheduledTrigger&nrtReplicas=3&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=2271
   [junit4]   2> 2120234 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.MiniSolrCloudCluster waitForActiveCollection: testScheduledTrigger
   [junit4]   2> 2120240 INFO  (qtp2099842840-25650) [n:127.0.0.1:65352_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testScheduledTrigger.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.testScheduledTrigger.shard1.replica_n4:INDEX.sizeInBytes} status=0 QTime=1
   [junit4]   2> 2120241 INFO  (qtp2099842840-25654) [n:127.0.0.1:65352_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2120242 INFO  (qtp1079919407-25651) [n:127.0.0.1:65351_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testScheduledTrigger.shard1.replica_n2:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 2120243 INFO  (qtp1079919407-25653) [n:127.0.0.1:65351_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 2120244 DEBUG (qtp2099842840-25652) [n:127.0.0.1:65352_solr     ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 2120245 INFO  (qtp2099842840-25652) [n:127.0.0.1:65352_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=8
   [junit4]   2> 2120246 DEBUG (zkCallback-8805-thread-1) [     ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 2120246 WARN  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 2120246 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 2120246 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 2120246 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 2120246 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.6+10
   [junit4]   2> 2120247 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread -- deactivating old nodeLost / nodeAdded markers
   [junit4]   2> 2120249 DEBUG (OverseerAutoScalingTriggerThread-72146898448154632-127.0.0.1:65351_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 2120249 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 2120249 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 2120249 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 2120249 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5fe4d91b{/solr,null,AVAILABLE}
   [junit4]   2> 2120250 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.s.AbstractConnector Started ServerConnector@486dedb4{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:65371}
   [junit4]   2> 2120250 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.s.Server Started @1974481ms
   [junit4]   2> 2120250 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65371}
   [junit4]   2> 2120250 ERROR (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2120250 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 2120251 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.6.0
   [junit4]   2> 2120251 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2120251 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr
   [junit4]   2> 2120251 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-03-16T17:04:57.479074Z
   [junit4]   2> 2120251 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2120254 INFO  (zkConnectionManagerCallback-8822-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2120254 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2120255 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2120440 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 2120441 WARN  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@df32331[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2120441 WARN  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@df32331[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2120444 WARN  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@63bb32e7[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2120444 WARN  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@63bb32e7[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 2120445 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65345/solr
   [junit4]   2> 2120446 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2120450 INFO  (zkConnectionManagerCallback-8829-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2120450 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2120560 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [n:127.0.0.1:65371_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2120564 INFO  (zkConnectionManagerCallback-8831-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2120564 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [n:127.0.0.1:65371_solr     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2120593 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [n:127.0.0.1:65371_solr     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2120617 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [n:127.0.0.1:65371_solr     ] o.a.s.c.ZkController Publish node=127.0.0.1:65371_solr as DOWN
   [junit4]   2> 2120625 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [n:127.0.0.1:65371_solr     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2120625 INFO  (TEST-ScheduledTriggerIntegrationTest.testScheduledTrigger-seed#[3FB101D065519D0B]) [n:127.0.0.1:65371_solr     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65371_solr
   [junit4]   2> 2120633 INFO  (zkCallback-8816-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2120633 INFO  (zkCallback-8805-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
 

[...truncated too long message...]

146930349572112-127.0.0.1:51825_solr-n_0000000002
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	42	/solr/live_nodes
   [junit4]   2> 	38	/solr/overseer/queue
   [junit4]   2> 	14	/solr/collections
   [junit4]   2> 	4	/solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> 2642781 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.c.ZkTestServer waitForServerDown: 127.0.0.1:51819
   [junit4]   2> 2642781 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:51819
   [junit4]   2> 2642781 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 51819
   [junit4]   2> 2642782 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[3FB101D065519D0B]-worker) [     ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Mar 16, 2020 5:13:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Mar 16, 2020 5:13:50 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest: 
   [junit4]   2>    1) Thread[id=30736, name=SessionTracker, state=TIMED_WAITING, group=TGRP-BlockJoinFacetDistribTest]
   [junit4]   2>         at java.base@11.0.6/java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)
   [junit4]   2> Mar 16, 2020 5:13:50 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=30736, name=SessionTracker, state=TIMED_WAITING, group=TGRP-BlockJoinFacetDistribTest]
   [junit4]   2> 2653184 ERROR (SessionTracker) [     ] o.a.z.s.ZooKeeperCriticalThread Severe unrecoverable error, from thread : SessionTracker
   [junit4]   2>           => java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.base/java.lang.Thread.sleep(Native Method)
   [junit4]   2> java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method) [?:?]
   [junit4]   2> 	at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) ~[zookeeper-3.5.7.jar:3.5.7]
   [junit4]   2> Mar 16, 2020 5:13:50 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFieldsData(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=1, maxDocsPerChunk=5, blockShift=7), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=1, blockSize=7)), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@38dae96), locale=sr-CS, timezone=America/North_Dakota/New_Salem
   [junit4]   2> NOTE: Mac OS X 10.14.6 x86_64/AdoptOpenJDK 11.0.6 (64-bit)/cpus=6,threads=1,free=157322704,total=387973120
   [junit4]   2> NOTE: All tests run in this JVM: [MoveReplicaHDFSFailoverTest, TestAtomicUpdateErrorCases, DistribCursorPagingTest, ClusterStateUpdateTest, Tagger2Test, NodeAddedTriggerIntegrationTest, TestCloudSearcherWarming, RestoreTriggerStateTest, FullHLLTest, SearchRateTriggerTest, TestTolerantUpdateProcessorRandomCloud, ExitableDirectoryReaderTest, TestSolr4Spatial, TestInPlaceUpdatesRequiredField, TriggerIntegrationTest, TestExactStatsCache, RuleEngineTest, LargeFieldTest, JWTVerificationkeyResolverTest, TestClassicSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestSubQueryTransformerCrossCore, TestDynamicURP, HDFSCollectionsAPITest, TestFieldCacheSortRandom, TestSolrQueryResponse, TestCustomSort, PathHierarchyTokenizerFactoryTest, BadComponentTest, TestManagedSynonymFilterFactory, OverseerModifyCollectionTest, ZkControllerTest, DistributedQueryComponentOptimizationTest, TestCursorMarkWithoutUniqueKey, TestLogWatcher, TestApiFramework, TestPKIAuthenticationPlugin, TestDocTermOrds, TestPullReplicaErrorHandling, TestReloadAndDeleteDocs, DistributedFacetSimpleRefinementLongTailTest, SimpleCollectionCreateDeleteTest, HttpPartitionOnCommitTest, TestCryptoKeys, BasicAuthOnSingleNodeTest, TestPHPSerializedResponseWriter, TestSimUtils, TestClusterStateMutator, ResponseBuilderTest, TestReqParamsAPI, NestedShardedAtomicUpdateTest, TestStressRecovery, AddReplicaTest, TestSkipOverseerOperations, TestSolrDeletionPolicy2, CollectionsAPIDistributedZkTest, SolrSlf4jReporterTest, TestCollationField, HdfsNNFailoverTest, TestCSVResponseWriter, OpenExchangeRatesOrgProviderTest, BlockCacheTest, RecoveryZkTest, AtomicUpdateProcessorFactoryTest, SolrCLIZkUtilsTest, RequestLoggingTest, TestCloudNestedDocsSort, TestReloadDeadlock, TermsComponentTest, DataDrivenBlockJoinTest, TestLegacyTerms, CoreAdminHandlerTest, TestSchemaSimilarityResource, TestBackupRepositoryFactory, TestNoOpRegenerator, HighlighterConfigTest, TestConfigSetImmutable, RequiredFieldsTest, FacetPivotSmallTest, StatsComponentTest, TestCorePropertiesReload, TestSolrConfigHandlerConcurrent, SynonymTokenizerTest, SystemCollectionCompatTest, OutputWriterTest, HdfsThreadLeakTest, TestIntervalFaceting, TestFunctionQuery, AnalysisAfterCoreReloadTest, BasicFunctionalityTest, ConvertedLegacyTest, CursorPagingTest, DisMaxRequestHandlerTest, DistributedIntervalFacetingTest, TestDistributedSearch, ChaosMonkeySafeLeaderTest, CollectionsAPISolrJTest, ReplicationFactorTest, TestRandomRequestDistribution, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestTlogReplayVsRecovery, TestWithCollection, HdfsAutoAddReplicasIntegrationTest, HttpTriggerListenerTest, IndexSizeTriggerSizeEstimationTest, TestSimLargeCluster, DistributedFacetExistsSmallTest, DistributedFacetPivotSmallTest, TestExportWriter, HighlighterWithoutStoredIdTest, TestUnifiedSolrHighlighterWithoutStoredId, SolrCoreMetricManagerTest, RegexBytesRefFilterTest, TestUnInvertedFieldException, TestWriterPerf, TestMacroExpander, TestMacros, JSONWriterTest, TestGeoJSONResponseWriter, TestRawResponseWriter, TestManagedResource, TestManagedResourceStorage, TestCopyFieldCollectionResource, TestDynamicFieldCollectionResource, TestFieldCollectionResource, TestFieldTypeCollectionResource, BadIndexSchemaTest, CurrencyFieldTypeTest, EnumFieldTest, IndexSchemaTest, TestManagedSchemaAPI, TestManagedSchemaThreadSafety, TestSchemaManager, TestSortableTextField, TestUseDocValuesAsStored2, QueryEqualityTest, QueryParsingTest, RankQueryTest, ReturnFieldsTest, SignificantTermsQParserPluginTest, SortSpecParsingTest, SpatialFilterTest, TestAddFieldRealTimeGet, TestCaffeineCache, TestGraphTermsQParserPlugin, TestIndexSearcher, TestPseudoReturnFields, TestQueryUtils, TestRangeQuery, TestSurroundQueryParser, TestValueSourceCache, TestJsonFacetsStatsParsing, TestOrdValues, TestSortByMinMaxFunction, DistanceFunctionTest, BJQParserTest, BlockJoinFacetDistribTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BlockJoinFacetDistribTest -Dtests.seed=3FB101D065519D0B -Dtests.slow=true -Dtests.locale=sr-CS -Dtests.timezone=America/North_Dakota/New_Salem -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J3 | BlockJoinFacetDistribTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest: 
   [junit4]    >    1) Thread[id=30736, name=SessionTracker, state=TIMED_WAITING, group=TGRP-BlockJoinFacetDistribTest]
   [junit4]    >         at java.base@11.0.6/java.lang.Thread.sleep(Native Method)
   [junit4]    >         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3FB101D065519D0B]:0)
   [junit4] Completed [779/901 (2!)] on J3 in 49.65s, 1 test, 1 error <<< FAILURES!

[...truncated 48354 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/750/consoleText

[repro] Revision: 33cd6909df28d39eedf2f777082ba26cf20a59ba

[repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC"
[repro] Repro line:  ant test  -Dtestcase=ScheduledTriggerIntegrationTest -Dtests.method=testScheduledTrigger -Dtests.seed=3FB101D065519D0B -Dtests.slow=true -Dtests.locale=dyo-SN -Dtests.timezone=Europe/Busingen -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=BlockJoinFacetDistribTest -Dtests.seed=3FB101D065519D0B -Dtests.slow=true -Dtests.locale=sr-CS -Dtests.timezone=America/North_Dakota/New_Salem -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] JUnit rest result XML files will be moved to: ./repro-reports
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       ScheduledTriggerIntegrationTest
[repro]       BlockJoinFacetDistribTest
[repro] ant compile-test

[...truncated 3448 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 -Dtests.class="*.ScheduledTriggerIntegrationTest|*.BlockJoinFacetDistribTest" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC" -Dtests.seed=3FB101D065519D0B -Dtests.slow=true -Dtests.locale=dyo-SN -Dtests.timezone=Europe/Busingen -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[...truncated 101 lines...]
[repro] Failures w/original seeds:
[repro]   0/5 failed: org.apache.solr.cloud.autoscaling.ScheduledTriggerIntegrationTest
[repro]   0/5 failed: org.apache.solr.search.join.BlockJoinFacetDistribTest
[repro] Exiting with code 0

[...truncated 73 lines...]