You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/06/06 23:55:32 UTC

[JENKINS-EA] Lucene-Solr-6.x-Linux (64bit/jdk-9-ea+121) - Build # 839 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/839/
Java: 64bit/jdk-9-ea+121 -XX:+UseCompressedOops -XX:+UseParallelGC

17 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
Error from server at http://127.0.0.1:45528/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1080)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:256)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(java.base@9-ea/Thread.java:843) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:809)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:468)  at java.util.concurrent.FutureTask.run(java.base@9-ea/FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1158)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45528/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1080)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:256)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:809)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:468)
	at java.util.concurrent.FutureTask.run(java.base@9-ea/FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1158)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([B721922F7358ADDD:3F75ADF5DDA4C025]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.tryDelete(ChaosMonkeySafeLeaderTest.java:190)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:92)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:531)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)


FAILED:  org.apache.solr.cloud.SimpleCollectionCreateDeleteTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([B721922F7358ADDD:3F75ADF5DDA4C025]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:718)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:427)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:332)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:990)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.LeaderElectionIntegrationTest

Error Message:
17 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionIntegrationTest:     1) Thread[id=1081, name=TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[B721922F7358ADDD]-SendThread(127.0.0.1:42177), state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    2) Thread[id=1060, name=Thread-389, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Object.wait(java.base@9-ea/Native Method)         at java.lang.Object.wait(java.base@9-ea/Object.java:516)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1253)    3) Thread[id=1054, name=TEST-LeaderElectionIntegrationTest.testLeaderElectionAfterClientTimeout-seed#[B721922F7358ADDD]-SendThread(127.0.0.1:34628), state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    4) Thread[id=1082, name=TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[B721922F7358ADDD]-EventThread, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.park(java.base@9-ea/LockSupport.java:190)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@9-ea/AbstractQueuedSynchronizer.java:2064)         at java.util.concurrent.LinkedBlockingQueue.take(java.base@9-ea/LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    5) Thread[id=1086, name=OverseerHdfsCoreFailoverThread-96026543810281475-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)    6) Thread[id=1059, name=OverseerHdfsCoreFailoverThread-96026522827489283-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)    7) Thread[id=1091, name=zkCallback-183-thread-4, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)    8) Thread[id=1057, name=OverseerStateUpdate-96026522827489283-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:418)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:145)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)    9) Thread[id=1085, name=OverseerCollectionConfigSetProcessor-96026543810281475-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:328)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:167)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)   10) Thread[id=1058, name=OverseerCollectionConfigSetProcessor-96026522827489283-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:328)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:167)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)   11) Thread[id=1087, name=Thread-391, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Object.wait(java.base@9-ea/Native Method)         at java.lang.Object.wait(java.base@9-ea/Object.java:516)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1253)   12) Thread[id=1093, name=zkCallback-183-thread-6, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)   13) Thread[id=1055, name=TEST-LeaderElectionIntegrationTest.testLeaderElectionAfterClientTimeout-seed#[B721922F7358ADDD]-EventThread, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.park(java.base@9-ea/LockSupport.java:190)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@9-ea/AbstractQueuedSynchronizer.java:2064)         at java.util.concurrent.LinkedBlockingQueue.take(java.base@9-ea/LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   14) Thread[id=1084, name=OverseerStateUpdate-96026543810281475-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:309)         at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:288)         at org.apache.solr.cloud.DistributedQueue.firstElement(DistributedQueue.java:357)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:164)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:142)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:196)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)   15) Thread[id=1089, name=zkCallback-183-thread-2, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)   16) Thread[id=1090, name=zkCallback-183-thread-3, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)   17) Thread[id=1092, name=zkCallback-183-thread-5, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 17 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionIntegrationTest: 
   1) Thread[id=1081, name=TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[B721922F7358ADDD]-SendThread(127.0.0.1:42177), state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   2) Thread[id=1060, name=Thread-389, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at java.lang.Object.wait(java.base@9-ea/Native Method)
        at java.lang.Object.wait(java.base@9-ea/Object.java:516)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1253)
   3) Thread[id=1054, name=TEST-LeaderElectionIntegrationTest.testLeaderElectionAfterClientTimeout-seed#[B721922F7358ADDD]-SendThread(127.0.0.1:34628), state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   4) Thread[id=1082, name=TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[B721922F7358ADDD]-EventThread, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.park(java.base@9-ea/LockSupport.java:190)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@9-ea/AbstractQueuedSynchronizer.java:2064)
        at java.util.concurrent.LinkedBlockingQueue.take(java.base@9-ea/LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   5) Thread[id=1086, name=OverseerHdfsCoreFailoverThread-96026543810281475-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   6) Thread[id=1059, name=OverseerHdfsCoreFailoverThread-96026522827489283-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   7) Thread[id=1091, name=zkCallback-183-thread-4, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   8) Thread[id=1057, name=OverseerStateUpdate-96026522827489283-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:418)
        at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:145)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   9) Thread[id=1085, name=OverseerCollectionConfigSetProcessor-96026543810281475-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:328)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:167)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
  10) Thread[id=1058, name=OverseerCollectionConfigSetProcessor-96026522827489283-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
        at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:328)
        at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:167)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
  11) Thread[id=1087, name=Thread-391, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at java.lang.Object.wait(java.base@9-ea/Native Method)
        at java.lang.Object.wait(java.base@9-ea/Object.java:516)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1253)
  12) Thread[id=1093, name=zkCallback-183-thread-6, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
  13) Thread[id=1055, name=TEST-LeaderElectionIntegrationTest.testLeaderElectionAfterClientTimeout-seed#[B721922F7358ADDD]-EventThread, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.park(java.base@9-ea/LockSupport.java:190)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@9-ea/AbstractQueuedSynchronizer.java:2064)
        at java.util.concurrent.LinkedBlockingQueue.take(java.base@9-ea/LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  14) Thread[id=1084, name=OverseerStateUpdate-96026543810281475-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
        at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
        at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:309)
        at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:288)
        at org.apache.solr.cloud.DistributedQueue.firstElement(DistributedQueue.java:357)
        at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:164)
        at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:142)
        at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:196)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
  15) Thread[id=1089, name=zkCallback-183-thread-2, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
  16) Thread[id=1090, name=zkCallback-183-thread-3, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
  17) Thread[id=1092, name=zkCallback-183-thread-5, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
	at __randomizedtesting.SeedInfo.seed([B721922F7358ADDD]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.LeaderElectionIntegrationTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1081, name=TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[B721922F7358ADDD]-SendThread(127.0.0.1:42177), state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    2) Thread[id=1087, name=Thread-391, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Object.wait(java.base@9-ea/Native Method)         at java.lang.Object.wait(java.base@9-ea/Object.java:516)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1253)    3) Thread[id=1093, name=zkCallback-183-thread-6, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)    4) Thread[id=1060, name=Thread-389, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Object.wait(java.base@9-ea/Native Method)         at java.lang.Object.wait(java.base@9-ea/Object.java:516)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1253)    5) Thread[id=1054, name=TEST-LeaderElectionIntegrationTest.testLeaderElectionAfterClientTimeout-seed#[B721922F7358ADDD]-SendThread(127.0.0.1:34628), state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    6) Thread[id=1086, name=OverseerHdfsCoreFailoverThread-96026543810281475-127.0.0.1:7000_solr-n_0000000000, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)    7) Thread[id=1059, name=OverseerHdfsCoreFailoverThread-96026522827489283-127.0.0.1:7000_solr-n_0000000000, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(java.base@9-ea/Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)    8) Thread[id=1091, name=zkCallback-183-thread-4, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)    9) Thread[id=1089, name=zkCallback-183-thread-2, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)   10) Thread[id=1090, name=zkCallback-183-thread-3, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)   11) Thread[id=1092, name=zkCallback-183-thread-5, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1081, name=TEST-LeaderElectionIntegrationTest.testSimpleSliceLeaderElection-seed#[B721922F7358ADDD]-SendThread(127.0.0.1:42177), state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   2) Thread[id=1087, name=Thread-391, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at java.lang.Object.wait(java.base@9-ea/Native Method)
        at java.lang.Object.wait(java.base@9-ea/Object.java:516)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1253)
   3) Thread[id=1093, name=zkCallback-183-thread-6, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   4) Thread[id=1060, name=Thread-389, state=WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at java.lang.Object.wait(java.base@9-ea/Native Method)
        at java.lang.Object.wait(java.base@9-ea/Object.java:516)
        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1253)
   5) Thread[id=1054, name=TEST-LeaderElectionIntegrationTest.testLeaderElectionAfterClientTimeout-seed#[B721922F7358ADDD]-SendThread(127.0.0.1:34628), state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   6) Thread[id=1086, name=OverseerHdfsCoreFailoverThread-96026543810281475-127.0.0.1:7000_solr-n_0000000000, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   7) Thread[id=1059, name=OverseerHdfsCoreFailoverThread-96026522827489283-127.0.0.1:7000_solr-n_0000000000, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(java.base@9-ea/Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   8) Thread[id=1091, name=zkCallback-183-thread-4, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   9) Thread[id=1089, name=zkCallback-183-thread-2, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
  10) Thread[id=1090, name=zkCallback-183-thread-3, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
  11) Thread[id=1092, name=zkCallback-183-thread-5, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
	at __randomizedtesting.SeedInfo.seed([B721922F7358ADDD]:0)


FAILED:  org.apache.solr.cloud.AliasIntegrationTest.test

Error Message:
Error from server at https://127.0.0.1:42951/lee/v/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /lee/v/collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1080)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:256)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(java.base@9-ea/Thread.java:843) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:809)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:468)  at java.util.concurrent.FutureTask.run(java.base@9-ea/FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1158)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:42951/lee/v/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /lee/v/collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1080)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:256)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:809)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:468)
	at java.util.concurrent.FutureTask.run(java.base@9-ea/FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1158)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([B721922F7358ADDD:3F75ADF5DDA4C025]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.AliasIntegrationTest.test(AliasIntegrationTest.java:69)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:531)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)


FAILED:  org.apache.solr.cloud.AsyncMigrateRouteKeyTest.test

Error Message:
Error from server at http://127.0.0.1:33488/at_jgc/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /at_jgc/collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1080)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:256)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(java.base@9-ea/Thread.java:843) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:809)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:468)  at java.util.concurrent.FutureTask.run(java.base@9-ea/FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1158)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33488/at_jgc/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /at_jgc/collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1080)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:256)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:809)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:468)
	at java.util.concurrent.FutureTask.run(java.base@9-ea/FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1158)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([B721922F7358ADDD:3F75ADF5DDA4C025]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.MigrateRouteKeyTest.multipleShardMigrateTest(MigrateRouteKeyTest.java:135)
	at org.apache.solr.cloud.AsyncMigrateRouteKeyTest.test(AsyncMigrateRouteKeyTest.java:44)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:531)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)


FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.test

Error Message:
Error from server at http://127.0.0.1:34084: Error CREATEing SolrCore 'onenodecollectioncore': Unable to create core [onenodecollectioncore] Caused by: Timed out getting coreNodeName for onenodecollectioncore

Stack Trace:
java.lang.AssertionError: Error from server at http://127.0.0.1:34084: Error CREATEing SolrCore 'onenodecollectioncore': Unable to create core [onenodecollectioncore] Caused by: Timed out getting coreNodeName for onenodecollectioncore
	at __randomizedtesting.SeedInfo.seed([B721922F7358ADDD:3F75ADF5DDA4C025]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:162)
	at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:71)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:531)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
Error from server at http://127.0.0.1:34798/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1080)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:256)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolr

[...truncated too long message...]

rDispatchFilter.java:208)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]    > 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
   [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
   [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
   [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]    > 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
   [junit4]    > 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    > 	at org.eclipse.jetty.server.Server.handle(Server.java:518)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   [junit4]    > 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
   [junit4]    > 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    > 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    > 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    > 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
   [junit4]    > 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]    > 	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
   [junit4]    > 	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
   [junit4]    > 	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
   [junit4]    > 	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:809)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:468)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(java.base@9-ea/FutureTask.java:266)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1158)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
   [junit4]    > 	... 1 more
   [junit4]    > ,code=500}</pre></p>
   [junit4]    > <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B721922F7358ADDD:3F75ADF5DDA4C025]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:128)
   [junit4]    > 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native Method)
   [junit4]    > 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
   [junit4]    > 	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 10414172 INFO  (SUITE-ChaosMonkeyNothingIsSafeTest-seed#[B721922F7358ADDD]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest_B721922F7358ADDD-001
   [junit4]   2> Jun 06, 2016 11:54:29 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, maxPointsInLeafNode=242, maxMBSortInHeap=5.517363650119625, sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=lu, timezone=Antarctica/Casey
   [junit4]   2> NOTE: Linux 4.4.0-21-generic amd64/Oracle Corporation 9-ea (64-bit)/cpus=12,threads=1,free=160036344,total=455606272
   [junit4]   2> NOTE: All tests run in this JVM: [TestQuerySenderListener, TestStressUserVersions, ChaosMonkeySafeLeaderTest, DistanceFunctionTest, TestScoreJoinQPNoScore, TestConfigSets, TestLegacyFieldCache, FileUtilsTest, TestQueryUtils, TestPivotHelperCode, TestSolr4Spatial2, TestUniqueKeyFieldResource, CollectionsAPIDistributedZkTest, TestChildDocTransformer, TestMiniSolrCloudClusterBase, SpellPossibilityIteratorTest, DirectSolrSpellCheckerTest, DeleteReplicaTest, HdfsWriteToMultipleCollectionsTest, ExitableDirectoryReaderTest, TestFieldTypeCollectionResource, DistributedQueueTest, TestTrie, OpenCloseCoreStressTest, SolrCmdDistributorTest, HighlighterMaxOffsetTest, TestSha256AuthenticationProvider, SharedFSAutoReplicaFailoverUtilsTest, TestLRUStatsCache, TestGeoJSONResponseWriter, HdfsRecoveryZkTest, UniqFieldsUpdateProcessorFactoryTest, HdfsBasicDistributedZkTest, TestAnalyzeInfixSuggestions, TestDistribIDF, AsyncMigrateRouteKeyTest, TestPartialUpdateDeduplication, TestSolrCoreProperties, MultiTermTest, TestFieldCacheReopen, TestDistributedGrouping, OutOfBoxZkACLAndCredentialsProvidersTest, TestCoreContainer, ShardRoutingCustomTest, CSVRequestHandlerTest, TestStressVersions, TestTestInjection, TestUseDocValuesAsStored2, SpellCheckComponentTest, AliasIntegrationTest, TestSubQueryTransformerDistrib, TestFiltering, TolerantUpdateProcessorTest, ZkStateReaderTest, DistributedIntervalFacetingTest, RestartWhileUpdatingTest, SolrXmlInZkTest, NotRequiredUniqueKeyTest, TestFreeTextSuggestions, TestTolerantUpdateProcessorCloud, SuggesterFSTTest, TestCollectionAPI, DistributedExpandComponentTest, SecurityConfHandlerTest, HttpPartitionTest, HLLSerializationTest, TestBlobHandler, TestConfigOverlay, CollectionReloadTest, SolrInfoMBeanTest, TestSurroundQueryParser, DistributedMLTComponentTest, TestManagedSchemaAPI, TestDynamicFieldResource, BlockDirectoryTest, DistributedQueryComponentCustomSortTest, TestFieldSortValues, RecoveryAfterSoftCommitTest, TestDownShardTolerantSearch, ShardRoutingTest, TestSolrDeletionPolicy2, TestCursorMarkWithoutUniqueKey, BadCopyFieldTest, DOMUtilTest, BasicFunctionalityTest, TestRestManager, TestCloudDeleteByQuery, TestReplicaProperties, TestSolrConfigHandlerCloud, TestInfoStreamLogging, HdfsCollectionsAPIDistributedZkTest, TestQuerySenderNoQuery, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed [607/613 (14!)] on J1 in 612.35s, 1 test, 1 error <<< FAILURES!

[...truncated 38 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:740: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:684: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:59: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build.xml:233: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/common-build.xml:531: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:1427: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:984: There were test failures: 613 suites (10 ignored), 2567 tests, 2 suite-level errors, 11 errors, 4 failures, 1777 ignored (63 assumptions) [seed: B721922F7358ADDD]

Total time: 197 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-6.x-Linux (64bit/jdk1.8.0_92) - Build # 840 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/840/
Java: 64bit/jdk1.8.0_92 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestCoreDiscovery

Error Message:
ObjectTracker found 4 object(s) that were not released!!! [MockDirectoryWrapper, MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [MockDirectoryWrapper, MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper]
	at __randomizedtesting.SeedInfo.seed([AE7EF232A10F4145]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestCoreDiscovery

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.core.TestCoreDiscovery:     1) Thread[id=8920, name=searcherExecutor-4505-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestCoreDiscovery: 
   1) Thread[id=8920, name=searcherExecutor-4505-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([AE7EF232A10F4145]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestCoreDiscovery

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=8920, name=searcherExecutor-4505-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=8920, name=searcherExecutor-4505-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([AE7EF232A10F4145]:0)




Build Log:
[...truncated 11968 lines...]
   [junit4] Suite: org.apache.solr.core.TestCoreDiscovery
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/init-core-data-001
   [junit4]   2> 1244388 INFO  (SUITE-TestCoreDiscovery-seed#[AE7EF232A10F4145]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1244388 INFO  (SUITE-TestCoreDiscovery-seed#[AE7EF232A10F4145]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 1244388 INFO  (SUITE-TestCoreDiscovery-seed#[AE7EF232A10F4145]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 1244389 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAlternateCoreDir
   [junit4]   2> 1244392 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244393 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-001
   [junit4]   2> 1244393 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-001'
   [junit4]   2> 1244393 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244393 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-001
   [junit4]   2> 1244393 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-001/solr.xml
   [junit4]   2> 1244396 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002
   [junit4]   2> 1244396 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CoreContainer New CoreContainer 1385185418
   [junit4]   2> 1244396 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-001]
   [junit4]   2> 1244396 WARN  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CoreContainer Couldn't add files from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-001/lib to classpath: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-001/lib
   [junit4]   2> 1244396 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1244398 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 1244399 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrCores Allocating transient cache for 2 transient cores
   [junit4]   2> 1244400 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1244400 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1244400 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1244400 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1244401 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002
   [junit4]   2> 1244401 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=core2, config=solrconfig-minimal.xml, loadOnStartup=false, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=false, dataDir=core2}
   [junit4]   2> 1244402 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CorePropertiesLocator Found core core2 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2
   [junit4]   2> 1244402 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=core1, config=solrconfig-minimal.xml, loadOnStartup=true, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=false, dataDir=core1}
   [junit4]   2> 1244402 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CorePropertiesLocator Found core core1 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1
   [junit4]   2> 1244402 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CorePropertiesLocator Found 2 core definitions
   [junit4]   2> 1244402 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1'
   [junit4]   2> 1244402 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244402 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-001
   [junit4]   2> 1244408 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1244410 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1244426 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 1244429 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.s.IndexSchema [core1] Schema name=tiny
   [junit4]   2> 1244432 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1244436 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1'
   [junit4]   2> 1244437 INFO  (coreLoadExecutor-4441-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'core1' using configuration from instancedir /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1
   [junit4]   2> 1244437 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1244437 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrCore [[core1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1/]
   [junit4]   2> 1244437 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrCore JMX monitoring not detected for core: core1
   [junit4]   2> 1244437 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1
   [junit4]   2> 1244437 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrCore New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1/index/
   [junit4]   2> 1244438 WARN  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrCore [core1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1/index' doesn't exist. Creating new index...
   [junit4]   2> 1244438 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1/index
   [junit4]   2> 1244438 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=38.2763671875, floorSegmentMB=2.154296875, forceMergeDeletesPctAllowed=1.3293771542077448, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1244438 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@381be64c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@27e00987),segFN=segments_1,generation=1}
   [junit4]   2> 1244438 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244439 INFO  (OldIndexDirectoryCleanupThreadForCore-core1) [    x:core1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core core1 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1/
   [junit4]   2> 1244439 WARN  (OldIndexDirectoryCleanupThreadForCore-core1) [    x:core1] o.a.s.c.DirectoryFactory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1244441 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1244443 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244443 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244444 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244444 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244445 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/graph,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1244445 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1244447 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1244447 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1244447 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6196029480165364]
   [junit4]   2> 1244447 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@381be64c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@27e00987),segFN=segments_1,generation=1}
   [junit4]   2> 1244447 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244448 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.s.SolrIndexSearcher Opening [Searcher@42a4a269[core1] main]
   [junit4]   2> 1244448 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/conf
   [junit4]   2> 1244448 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/conf}
   [junit4]   2> 1244448 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/conf
   [junit4]   2> 1244448 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1244448 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1244448 INFO  (searcherExecutor-4442-thread-1-processing-x:core1) [    x:core1] o.a.s.c.SolrCore [core1] Registered new searcher Searcher@42a4a269[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1244448 INFO  (coreLoadExecutor-4441-thread-1) [    x:core1] o.a.s.c.CoreContainer registering core: core1
   [junit4]   2> 1244448 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2'
   [junit4]   2> 1244449 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244449 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-001
   [junit4]   2> 1244454 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1244457 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1244462 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 1244466 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.s.IndexSchema [core2] Schema name=tiny
   [junit4]   2> 1244468 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1244469 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2'
   [junit4]   2> 1244469 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    ] o.a.s.c.CoreContainer Creating SolrCore 'core2' using configuration from instancedir /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2
   [junit4]   2> 1244469 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1244469 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrCore [[core2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2/]
   [junit4]   2> 1244469 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrCore JMX monitoring not detected for core: core2
   [junit4]   2> 1244469 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2
   [junit4]   2> 1244469 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrCore New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2/index/
   [junit4]   2> 1244470 WARN  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrCore [core2] Solr index directory '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2/index' doesn't exist. Creating new index...
   [junit4]   2> 1244470 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2/index
   [junit4]   2> 1244470 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=21, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1244470 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@2ce41f44 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36bd269c),segFN=segments_1,generation=1}
   [junit4]   2> 1244470 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244470 INFO  (OldIndexDirectoryCleanupThreadForCore-core2) [    x:core2] o.a.s.c.SolrCore Looking for old index directories to cleanup for core core2 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2/
   [junit4]   2> 1244471 WARN  (OldIndexDirectoryCleanupThreadForCore-core2) [    x:core2] o.a.s.c.DirectoryFactory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1244472 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1244473 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244473 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244473 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244474 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244475 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/graph,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1244475 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1244475 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1244476 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1244476 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=77.130859375, floorSegmentMB=0.486328125, forceMergeDeletesPctAllowed=29.512037338187863, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1244476 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@2ce41f44 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36bd269c),segFN=segments_1,generation=1}
   [junit4]   2> 1244476 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244476 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.s.SolrIndexSearcher Opening [Searcher@37c84115[core2] main]
   [junit4]   2> 1244477 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/conf
   [junit4]   2> 1244477 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/conf}
   [junit4]   2> 1244477 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/conf
   [junit4]   2> 1244477 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1244477 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1244477 INFO  (searcherExecutor-4446-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrCore [core2] Registered new searcher Searcher@37c84115[core2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1244477 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer registering core: core2
   [junit4]   2> 1244478 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1385185418
   [junit4]   2> 1244478 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.c.SolrCore [core2]  CLOSING SolrCore org.apache.solr.core.SolrCore@33e9c1e2
   [junit4]   2> 1244478 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.u.DirectUpdateHandler2 closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1244478 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.u.SolrCoreState Closing SolrCoreState
   [junit4]   2> 1244478 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1244478 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.u.DefaultSolrCoreState closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1244479 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.c.SolrCore [core2] Closing main searcher on request.
   [junit4]   2> 1244479 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1244479 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.c.CachingDirectoryFactory looking to close /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2 [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2;done=false>>]
   [junit4]   2> 1244479 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 1244479 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.c.CachingDirectoryFactory Closing directory: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2
   [junit4]   2> 1244479 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.c.CachingDirectoryFactory looking to close /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2/index [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2/index;done=false>>]
   [junit4]   2> 1244479 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 1244479 INFO  (coreCloseExecutor-4450-thread-2-processing-x:core2) [    x:core2] o.a.s.c.CachingDirectoryFactory Closing directory: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core2/core2/index
   [junit4]   2> 1244494 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.c.SolrCore [core1]  CLOSING SolrCore org.apache.solr.core.SolrCore@440e8e7f
   [junit4]   2> 1244494 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.u.DirectUpdateHandler2 closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1244494 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.u.SolrCoreState Closing SolrCoreState
   [junit4]   2> 1244494 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1244494 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.u.DefaultSolrCoreState closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1244495 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.c.SolrCore [core1] Closing main searcher on request.
   [junit4]   2> 1244495 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1244495 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.c.CachingDirectoryFactory looking to close /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1 [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1;done=false>>]
   [junit4]   2> 1244495 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 1244495 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.c.CachingDirectoryFactory Closing directory: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1
   [junit4]   2> 1244495 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.c.CachingDirectoryFactory looking to close /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1/index [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1/index;done=false>>]
   [junit4]   2> 1244495 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 1244495 INFO  (coreCloseExecutor-4450-thread-1-processing-x:core2) [    x:core1] o.a.s.c.CachingDirectoryFactory Closing directory: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-002/core1/core1/index
   [junit4]   2> 1244495 INFO  (TEST-TestCoreDiscovery.testAlternateCoreDir-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.SolrTestCaseJ4 ###Ending testAlternateCoreDir
   [junit4]   2> 1244497 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.SolrTestCaseJ4 ###Starting testDuplicateNames
   [junit4]   2> 1244500 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244501 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003
   [junit4]   2> 1244501 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003'
   [junit4]   2> 1244501 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244501 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003
   [junit4]   2> 1244501 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003/solr.xml
   [junit4]   2> 1244503 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003
   [junit4]   2> 1244503 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer New CoreContainer 511314936
   [junit4]   2> 1244503 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003]
   [junit4]   2> 1244504 WARN  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer Couldn't add files from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003/lib to classpath: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003/lib
   [junit4]   2> 1244504 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1244504 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 1244505 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrCores Allocating transient cache for 2 transient cores
   [junit4]   2> 1244505 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1244505 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1244505 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1244505 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1244505 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003
   [junit4]   2> 1244505 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=core1, config=solrconfig-minimal.xml, loadOnStartup=false, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=false, dataDir=stuffandnonsense}
   [junit4]   2> 1244506 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found core core1 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003/core2
   [junit4]   2> 1244506 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=core1, config=solrconfig-minimal.xml, loadOnStartup=true, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=false, dataDir=stuffandnonsense}
   [junit4]   2> 1244506 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found core core1 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-003/core1
   [junit4]   2> 1244506 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found 2 core definitions
   [junit4]   2> 1244506 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer Shutting down CoreContainer instance=511314936
   [junit4]   2> 1244506 INFO  (TEST-TestCoreDiscovery.testDuplicateNames-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.SolrTestCaseJ4 ###Ending testDuplicateNames
   [junit4]   2> 1244508 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.SolrTestCaseJ4 ###Starting testTooManyTransientCores
   [junit4]   2> 1244516 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004'
   [junit4]   2> 1244516 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244516 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004
   [junit4]   2> 1244516 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/solr.xml
   [junit4]   2> 1244519 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004
   [junit4]   2> 1244519 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer New CoreContainer 243489109
   [junit4]   2> 1244519 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004]
   [junit4]   2> 1244519 WARN  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer Couldn't add files from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/lib to classpath: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/lib
   [junit4]   2> 1244519 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1244520 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 1244521 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.SolrCores Allocating transient cache for 2 transient cores
   [junit4]   2> 1244521 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1244521 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1244521 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1244521 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1244521 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004
   [junit4]   2> 1244522 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=coreT5, config=solrconfig-minimal.xml, loadOnStartup=true, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=coreT5}
   [junit4]   2> 1244522 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found core coreT5 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5
   [junit4]   2> 1244522 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=coreT1, config=solrconfig-minimal.xml, loadOnStartup=true, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=coreT1}
   [junit4]   2> 1244522 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found core coreT1 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1
   [junit4]   2> 1244522 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=coreLOS, config=solrconfig-minimal.xml, loadOnStartup=true, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=false, dataDir=coreLOS}
   [junit4]   2> 1244522 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found core coreLOS in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS
   [junit4]   2> 1244523 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=coreT4, config=solrconfig-minimal.xml, loadOnStartup=true, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=coreT4}
   [junit4]   2> 1244523 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found core coreT4 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT4
   [junit4]   2> 1244523 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=coreT2, config=solrconfig-minimal.xml, loadOnStartup=true, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=coreT2}
   [junit4]   2> 1244523 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found core coreT2 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT2
   [junit4]   2> 1244524 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=coreT6, config=solrconfig-minimal.xml, loadOnStartup=true, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=coreT6}
   [junit4]   2> 1244524 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found core coreT6 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT6
   [junit4]   2> 1244524 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=coreT3, config=solrconfig-minimal.xml, loadOnStartup=true, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=coreT3}
   [junit4]   2> 1244524 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found core coreT3 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT3
   [junit4]   2> 1244524 INFO  (TEST-TestCoreDiscovery.testTooManyTransientCores-seed#[AE7EF232A10F4145]) [    x:core2] o.a.s.c.CorePropertiesLocator Found 7 core definitions
   [junit4]   2> 1244525 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5'
   [junit4]   2> 1244525 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244525 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004
   [junit4]   2> 1244529 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1'
   [junit4]   2> 1244529 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS'
   [junit4]   2> 1244529 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244529 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004
   [junit4]   2> 1244529 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244529 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004
   [junit4]   2> 1244533 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1244533 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1244534 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1244534 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1244549 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1244552 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 1244554 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.s.IndexSchema [coreT1] Schema name=tiny
   [junit4]   2> 1244556 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1244557 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1'
   [junit4]   2> 1244557 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.c.CoreContainer Creating SolrCore 'coreT1' using configuration from instancedir /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1
   [junit4]   2> 1244557 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1244558 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrCore [[coreT1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/coreT1/]
   [junit4]   2> 1244558 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrCore JMX monitoring not detected for core: coreT1
   [junit4]   2> 1244558 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/coreT1
   [junit4]   2> 1244558 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1244562 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 1244563 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 1244564 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrCore New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/coreT1/index/
   [junit4]   2> 1244564 WARN  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrCore [coreT1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/coreT1/index' doesn't exist. Creating new index...
   [junit4]   2> 1244564 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/coreT1/index
   [junit4]   2> 1244564 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=38.2763671875, floorSegmentMB=2.154296875, forceMergeDeletesPctAllowed=1.3293771542077448, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1244565 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@532e2ba lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@26edd96f),segFN=segments_1,generation=1}
   [junit4]   2> 1244565 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244565 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.s.IndexSchema [coreT5] Schema name=tiny
   [junit4]   2> 1244565 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.s.IndexSchema [coreLOS] Schema name=tiny
   [junit4]   2> 1244567 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1244567 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1244567 INFO  (OldIndexDirectoryCleanupThreadForCore-coreT1) [    x:coreT1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core coreT1 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/coreT1/
   [junit4]   2> 1244567 WARN  (OldIndexDirectoryCleanupThreadForCore-coreT1) [    x:coreT1] o.a.s.c.DirectoryFactory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/coreT1/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1244568 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5'
   [junit4]   2> 1244568 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.CoreContainer Creating SolrCore 'coreT5' using configuration from instancedir /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5
   [junit4]   2> 1244568 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS'
   [junit4]   2> 1244568 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:core2] o.a.s.c.CoreContainer Creating SolrCore 'coreLOS' using configuration from instancedir /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS
   [junit4]   2> 1244568 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1244568 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1244568 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrCore [[coreT5] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/coreT5/]
   [junit4]   2> 1244568 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrCore [[coreLOS] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/coreLOS/]
   [junit4]   2> 1244568 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrCore JMX monitoring not detected for core: coreT5
   [junit4]   2> 1244569 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrCore JMX monitoring not detected for core: coreLOS
   [junit4]   2> 1244569 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/coreT5
   [junit4]   2> 1244569 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/coreLOS
   [junit4]   2> 1244569 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrCore New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/coreT5/index/
   [junit4]   2> 1244569 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrCore New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/coreLOS/index/
   [junit4]   2> 1244569 WARN  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrCore [coreT5] Solr index directory '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/coreT5/index' doesn't exist. Creating new index...
   [junit4]   2> 1244569 WARN  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrCore [coreLOS] Solr index directory '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/coreLOS/index' doesn't exist. Creating new index...
   [junit4]   2> 1244569 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/coreT5/index
   [junit4]   2> 1244570 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/coreLOS/index
   [junit4]   2> 1244570 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=38.2763671875, floorSegmentMB=2.154296875, forceMergeDeletesPctAllowed=1.3293771542077448, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1244570 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=38.2763671875, floorSegmentMB=2.154296875, forceMergeDeletesPctAllowed=1.3293771542077448, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1244570 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6071c9f3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e2fcf84),segFN=segments_1,generation=1}
   [junit4]   2> 1244570 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244570 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7e01285f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e5eced1),segFN=segments_1,generation=1}
   [junit4]   2> 1244570 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244570 INFO  (OldIndexDirectoryCleanupThreadForCore-coreT5) [    x:coreT5] o.a.s.c.SolrCore Looking for old index directories to cleanup for core coreT5 in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/coreT5/
   [junit4]   2> 1244570 WARN  (OldIndexDirectoryCleanupThreadForCore-coreT5) [    x:coreT5] o.a.s.c.DirectoryFactory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/coreT5/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1244571 INFO  (OldIndexDirectoryCleanupThreadForCore-coreLOS) [    x:coreLOS] o.a.s.c.SolrCore Looking for old index directories to cleanup for core coreLOS in /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/coreLOS/
   [junit4]   2> 1244571 WARN  (OldIndexDirectoryCleanupThreadForCore-coreLOS) [    x:coreLOS] o.a.s.c.DirectoryFactory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/coreLOS/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1244572 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1244572 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1244572 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1244573 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244573 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244573 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244574 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244574 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244574 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244574 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244575 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244575 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244575 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/graph,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6196029480165364]
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6071c9f3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e2fcf84),segFN=segments_1,generation=1}
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244576 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.s.SolrIndexSearcher Opening [Searcher@6a3f5db8[coreT5] main]
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/conf
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/conf}
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/graph,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT5/conf
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/graph,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1244577 INFO  (searcherExecutor-4466-thread-1-processing-x:coreT5) [    x:coreT5] o.a.s.c.SolrCore [coreT5] Registered new searcher Searcher@6a3f5db8[coreT5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6196029480165364]
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.SolrCores Opening transient core coreT5
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:coreT5] o.a.s.c.CoreContainer registering core: coreT5
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@532e2ba lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@26edd96f),segFN=segments_1,generation=1}
   [junit4]   2> 1244577 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT4'
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6196029480165364]
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7e01285f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e5eced1),segFN=segments_1,generation=1}
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.s.SolrIndexSearcher Opening [Searcher@154d1d1c[coreT1] main]
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-1-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/conf
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.s.SolrIndexSearcher Opening [Searcher@58a9e044[coreLOS] main]
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/conf}
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreT1/conf
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/conf
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/conf}
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.core.TestCoreDiscovery_AE7EF232A10F4145-001/tempDir-004/coreLOS/conf
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1244578 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1244579 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1244579 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1244579 INFO  (searcherExecutor-4465-thread-1-processing-x:coreT1) [    x:coreT1] o.a.s.c.SolrCore [coreT1] Registered new searcher Searcher@154d1d1c[coreT1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1244579 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.SolrCores Opening transient core coreT1
   [junit4]   2> 1244579 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:coreT1] o.a.s.c.CoreContainer registering core: coreT1
   [junit4]   2> 1244579 INFO  (coreLoadExecutor-4464-thread-3-processing-x:core2) [    x:coreLOS] o.a.s.c.CoreContainer registering core: coreLOS
   [junit4]   2> 1244579 INFO  (coreLoadExecutor-4464-thread-2-processing-x:core2) [    x:core2] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspa

[...truncated too long message...]

Thread.java:745)
   [junit4]   2> 
   [junit4]   2> Jun 07, 2016 2:17:23 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Jun 07, 2016 2:17:43 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.core.TestCoreDiscovery: 
   [junit4]   2>    1) Thread[id=8920, name=searcherExecutor-4505-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Jun 07, 2016 2:17:43 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=8920, name=searcherExecutor-4505-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
   [junit4]   2> Jun 07, 2016 2:17:46 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=8920, name=searcherExecutor-4505-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, locale=et, timezone=America/Regina
   [junit4]   2> NOTE: Linux 4.4.0-21-generic amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=12,threads=2,free=116467560,total=518148096
   [junit4]   2> NOTE: All tests run in this JVM: [NoCacheHeaderTest, TestSolrQueryParser, TestCryptoKeys, PreAnalyzedUpdateProcessorTest, TestPerFieldSimilarity, ZkStateWriterTest, CleanupOldIndexTest, TestConfigSetImmutable, MergeStrategyTest, TestExactSharedStatsCache, TestInitParams, TestSolrIndexConfig, BJQParserTest, HdfsNNFailoverTest, TestFuzzyAnalyzedSuggestions, StatelessScriptUpdateProcessorFactoryTest, HighlighterConfigTest, HardAutoCommitTest, ShowFileRequestHandlerTest, TestSolrXml, SuggesterTest, TestReload, TestScoreJoinQPScore, TestRestoreCore, TestDistributedMissingSort, ResponseHeaderTest, TestShardHandlerFactory, TestFieldTypeResource, SegmentsInfoRequestHandlerTest, CdcrRequestHandlerTest, TestPKIAuthenticationPlugin, TestSolrDeletionPolicy1, TermVectorComponentDistributedTest, DirectSolrConnectionTest, BasicAuthIntegrationTest, PrimitiveFieldTypeTest, BlockJoinFacetSimpleTest, BigEndianAscendingWordDeserializerTest, LukeRequestHandlerTest, TestCollationField, TestFastWriter, ShardSplitTest, LeaderElectionTest, TestCloudInspectUtil, TestSchemaSimilarityResource, AtomicUpdatesTest, TestLockTree, TestRandomCollapseQParserPlugin, TestStressCloudBlindAtomicUpdates, XmlUpdateRequestHandlerTest, TestSweetSpotSimilarityFactory, SpatialHeatmapFacetsTest, OverseerStatusTest, TestRandomRequestDistribution, TestTolerantSearch, TestRecovery, DistributedFacetPivotLargeTest, TestSolrQueryParserDefaultOperatorResource, TestZkChroot, TestSort, StatsComponentTest, HdfsChaosMonkeyNothingIsSafeTest, TestConfigSetsAPI, AssignTest, BadComponentTest, TestIndexingPerformance, DistributedFacetPivotSmallTest, RemoteQueryErrorTest, TestPostingsSolrHighlighter, TestReversedWildcardFilterFactory, TestHdfsUpdateLog, TestIndexSearcher, TestRTimerTree, TestWriterPerf, RegexBoostProcessorTest, MigrateRouteKeyTest, QueryParsingTest, TestComplexPhraseQParserPlugin, TestJmxMonitoredMap, RecoveryZkTest, TestRequestForwarding, UUIDUpdateProcessorFallbackTest, PluginInfoTest, SimpleCollectionCreateDeleteTest, HLLUtilTest, TestSolrQueryResponse, HdfsTlogReplayBufferedWhileIndexingTest, TestNumericTerms32, TestGroupingSearch, TestBulkSchemaAPI, EnumFieldTest, DistributedVersionInfoTest, TestTolerantUpdateProcessorRandomCloud, DistribDocExpirationUpdateProcessorTest, DebugComponentTest, RequestHandlersTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestDistributedSearch, DistributedSpellCheckComponentTest, TestStressVersions, SimpleFacetsTest, TestSolr4Spatial, SolrCmdDistributorTest, TestFunctionQuery, BasicFunctionalityTest, DirectUpdateHandlerTest, TestBadConfig, SoftAutoCommitTest, SimplePostToolTest, TestCoreDiscovery]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCoreDiscovery -Dtests.seed=AE7EF232A10F4145 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=et -Dtests.timezone=America/Regina -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestCoreDiscovery (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [MockDirectoryWrapper, MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE7EF232A10F4145]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestCoreDiscovery: 
   [junit4]    >    1) Thread[id=8920, name=searcherExecutor-4505-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE7EF232A10F4145]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=8920, name=searcherExecutor-4505-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE7EF232A10F4145]:0)
   [junit4] Completed [446/613 (1!)] on J2 in 145.29s, 12 tests, 1 failure, 2 errors <<< FAILURES!

[...truncated 551 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:740: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:684: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:59: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build.xml:233: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/common-build.xml:531: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:1427: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:984: There were test failures: 613 suites (10 ignored), 2567 tests, 3 suite-level errors, 682 ignored (67 assumptions) [seed: AE7EF232A10F4145]

Total time: 48 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any