You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2017/12/08 17:32:11 UTC

[JENKINS] Lucene-Solr-Tests-7.x - Build # 276 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/276/

All tests passed

Build Log:
[...truncated 466 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J0-20171208_165844_8742006765338077402132.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] #
   [junit4] # There is insufficient memory for the Java Runtime Environment to continue.
   [junit4] # Native memory allocation (mmap) failed to map 114294784 bytes for committing reserved memory.
   [junit4] # An error report file with more information is saved as:
   [junit4] # /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/J0/hs_err_pid24683.log
   [junit4] <<< JVM J0: EOF ----

   [junit4] JVM J0: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/lucene/build/core/test/temp/junit4-J0-20171208_165844_8749039779944601907095.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000eca00000, 114294784, 0) failed; error='Cannot allocate memory' (errno=12)
   [junit4] <<< JVM J0: EOF ----

[...truncated 824 lines...]
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 142606336 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/hs_err_pid24332.log
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-Tests-7.x #274 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-Tests-7.x - Build # 286 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/286/

9 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Expected to find shardAddress in the up shard info: {error=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded,trace=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded  at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:460)  at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)  at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748) ,time=2}

Stack Trace:
java.lang.AssertionError: Expected to find shardAddress in the up shard info: {error=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded,trace=org.apache.solr.client.solrj.SolrServerException: Time allowed to handle this request exceeded
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:460)
	at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273)
	at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
,time=2}
	at __randomizedtesting.SeedInfo.seed([E7C582025295575F:6F91BDD8FC693AA7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.TestDistributedSearch.comparePartialResponses(TestDistributedSearch.java:1191)
	at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1132)
	at org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:992)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1019)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards

Error Message:
Error from server at https://127.0.0.1:59452/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000010

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:59452/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000010
	at __randomizedtesting.SeedInfo.seed([E7C582025295575F:DB54C5561DD40A0A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaByCountForAllShards(DeleteReplicaTest.java:136)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored_idx

Error Message:
java.lang.RuntimeException: Error from server at http://127.0.0.1:54340/solr/test_col: Cannot talk to ZooKeeper - Updates are disabled.

Stack Trace:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error from server at http://127.0.0.1:54340/solr/test_col: Cannot talk to ZooKeeper - Updates are disabled.
	at __randomizedtesting.SeedInfo.seed([E7C582025295575F:EDE2D3222DE3F408]:0)
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:283)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored_idx(TestStressCloudBlindAtomicUpdates.java:216)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Error from server at http://127.0.0.1:54340/solr/test_col: Cannot talk to ZooKeeper - Updates are disabled.
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:411)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:54340/solr/test_col: Cannot talk to ZooKeeper - Updates are disabled.
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.doRandomAtomicUpdate(TestStressCloudBlindAtomicUpdates.java:370)
	at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:399)
	... 6 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTlogReplica

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:351)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:729)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:954)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1039)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:498)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:948)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1039)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:761)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:954)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1039)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1039)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)  at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1039)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)  at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)  at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)  at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)  at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:534)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:501)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)  at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1159)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:351)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:729)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:954)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1039)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:534)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
	at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:498)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:948)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1039)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:534)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
	at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:761)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:954)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1039)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:534)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1039)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:863)
	at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1039)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:950)
	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:534)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:501)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420)
	at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1159)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


	at __randomizedtesting.SeedInfo.seed([E7C582025295575F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:297)
	at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testCooldown

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([E7C582025295575F:D67BEFE62C3F22AD]:0)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testCooldown(TestTriggerIntegration.java:1084)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeLostTriggerRestoreState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
	at __randomizedtesting.SeedInfo.seed([E7C582025295575F:CC3A5759C8ED428F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeLostTriggerRestoreState(TestTriggerIntegration.java:307)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.TestSolrConfigHandlerCloud.test

Error Message:
{   "responseHeader":{     "status":500,     "QTime":31789},   "errorMessages":["4 out of 5 the property overlay to be of version 10 within 30 seconds! Failed cores: [https://127.0.0.1:42244/c_h/qt/collection1_shard2_replica_n41/, https://127.0.0.1:49332/c_h/qt/collection1_shard2_replica_n45/, https://127.0.0.1:39168/c_h/qt/collection1_shard1_replica_n43/, https://127.0.0.1:60638/c_h/qt/collection1_shard1_replica_n47/]\n"],   "WARNING":"This response format is experimental.  It is likely to change in the future.",   "error":{     "metadata":[       "error-class","org.apache.solr.common.SolrException",       "root-error-class","org.apache.solr.common.SolrException"],     "msg":"4 out of 5 the property overlay to be of version 10 within 30 seconds! Failed cores: [https://127.0.0.1:42244/c_h/qt/collection1_shard2_replica_n41/, https://127.0.0.1:49332/c_h/qt/collection1_shard2_replica_n45/, https://127.0.0.1:39168/c_h/qt/collection1_shard1_replica_n43/, https://127.0.0.1:60638/c_h/qt/collection1_shard1_replica_n47/]",     "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 10 within 30 seconds! Failed cores: [https://127.0.0.1:42244/c_h/qt/collection1_shard2_replica_n41/, https://127.0.0.1:49332/c_h/qt/collection1_shard2_replica_n45/, https://127.0.0.1:39168/c_h/qt/collection1_shard1_replica_n43/, https://127.0.0.1:60638/c_h/qt/collection1_shard1_replica_n47/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",     "code":500}} 

Stack Trace:
java.lang.AssertionError: {
  "responseHeader":{
    "status":500,
    "QTime":31789},
  "errorMessages":["4 out of 5 the property overlay to be of version 10 within 30 seconds! Failed cores: [https://127.0.0.1:42244/c_h/qt/collection1_shard2_replica_n41/, https://127.0.0.1:49332/c_h/qt/collection1_shard2_replica_n45/, https://127.0.0.1:39168/c_h/qt/collection1_shard1_replica_n43/, https://127.0.0.1:60638/c_h/qt/collection1_shard1_replica_n47/]\n"],
  "WARNING":"This response format is experimental.  It is likely to change in the future.",
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"4 out of 5 the property overlay to be of version 10 within 30 seconds! Failed cores: [https://127.0.0.1:42244/c_h/qt/collection1_shard2_replica_n41/, https://127.0.0.1:49332/c_h/qt/collection1_shard2_replica_n45/, https://127.0.0.1:39168/c_h/qt/collection1_shard1_replica_n43/, https://127.0.0.1:60638/c_h/qt/collection1_shard1_replica_n47/]",
    "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 10 within 30 seconds! Failed cores: [https://127.0.0.1:42244/c_h/qt/collection1_shard2_replica_n41/, https://127.0.0.1:49332/c_h/qt/collection1_shard2_replica_n45/, https://127.0.0.1:39168/c_h/qt/collection1_shard1_replica_n43/, https://127.0.0.1:60638/c_h/qt/collection1_shard1_replica_n47/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",
    "code":500}}

	at __randomizedtesting.SeedInfo.seed([E7C582025295575F:6F91BDD8FC693AA7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.core.TestSolrConfigHandler.runConfigCommand(TestSolrConfigHandler.java:185)
	at org.apache.solr.core.TestSolrConfigHandler.reqhandlertests(TestSolrConfigHandler.java:342)
	at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqHandlerAPIs(TestSolrConfigHandlerCloud.java:97)
	at org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.schema.TestCloudSchemaless.test

Error Message:
Error from server at http://127.0.0.1:45079/collection1: 4 Async exceptions during distributed update:  Error from server at http://127.0.0.1:38863/collection1_shard3_replica_n83: Server Error    request: http://127.0.0.1:38863/collection1_shard3_replica_n83/update?update.chain=add-unknown-fields-to-the-schema&update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A45079%2Fcollection1_shard3_replica_n91%2F&wt=javabin&version=2 Remote error message: Error persisting managed schema at /configs/conf1/managed-schema Error from server at http://127.0.0.1:56557/collection1_shard1_replica_n87: Server Error    request: http://127.0.0.1:56557/collection1_shard1_replica_n87/update?update.chain=add-unknown-fields-to-the-schema&update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A45079%2Fcollection1_shard3_replica_n91%2F&wt=javabin&version=2 Remote error message: Error persisting managed schema at /configs/conf1/managed-schema Error from server at http://127.0.0.1:57621/collection1_shard4_replica_n85: Server Error    request: http://127.0.0.1:57621/collection1_shard4_replica_n85/update?update.chain=add-unknown-fields-to-the-schema&update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A45079%2Fcollection1_shard3_replica_n91%2F&wt=javabin&version=2 Remote error message: Error persisting managed schema at /configs/conf1/managed-schema Error from server at http://127.0.0.1:48339/collection1_shard2_replica_n81: Server Error    request: http://127.0.0.1:48339/collection1_shard2_replica_n81/update?update.chain=add-unknown-fields-to-the-schema&update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A45079%2Fcollection1_shard3_replica_n91%2F&wt=javabin&version=2 Remote error message: Error persisting managed schema at /configs/conf1/managed-schema

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45079/collection1: 4 Async exceptions during distributed update: 
Error from server at http://127.0.0.1:38863/collection1_shard3_replica_n83: Server Error



request: http://127.0.0.1:38863/collection1_shard3_replica_n83/update?update.chain=add-unknown-fields-to-the-schema&update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A45079%2Fcollection1_shard3_replica_n91%2F&wt=javabin&version=2
Remote error message: Error persisting managed schema at /configs/conf1/managed-schema
Error from server at http://127.0.0.1:56557/collection1_shard1_replica_n87: Server Error



request: http://127.0.0.1:56557/collection1_shard1_replica_n87/update?update.chain=add-unknown-fields-to-the-schema&update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A45079%2Fcollection1_shard3_replica_n91%2F&wt=javabin&version=2
Remote error message: Error persisting managed schema at /configs/conf1/managed-schema
Error from server at http://127.0.0.1:57621/collection1_shard4_replica_n85: Server Error



request: http://127.0.0.1:57621/collection1_shard4_replica_n85/update?update.chain=add-unknown-fields-to-the-schema&update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A45079%2Fcollection1_shard3_replica_n91%2F&wt=javabin&version=2
Remote error message: Error persisting managed schema at /configs/conf1/managed-schema
Error from server at http://127.0.0.1:48339/collection1_shard2_replica_n81: Server Error



request: http://127.0.0.1:48339/collection1_shard2_replica_n81/update?update.chain=add-unknown-fields-to-the-schema&update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A45079%2Fcollection1_shard3_replica_n91%2F&wt=javabin&version=2
Remote error message: Error persisting managed schema at /configs/conf1/managed-schema
	at __randomizedtesting.SeedInfo.seed([E7C582025295575F:6F91BDD8FC693AA7]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
	at org.apache.solr.schema.TestCloudSchemaless.test(TestCloudSchemaless.java:124)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:38825_solr wasn't created

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:38825_solr wasn't created
	at __randomizedtesting.SeedInfo.seed([E7C582025295575F:FF7F0A0E5CA09AB0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration(TriggerIntegrationTest.java:932)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12454 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_E7C582025295575F-001/init-core-data-001
   [junit4]   2> 2951227 WARN  (SUITE-TestTriggerIntegration-seed#[E7C582025295575F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2951235 INFO  (SUITE-TestTriggerIntegration-seed#[E7C582025295575F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2951236 INFO  (SUITE-TestTriggerIntegration-seed#[E7C582025295575F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2951236 INFO  (SUITE-TestTriggerIntegration-seed#[E7C582025295575F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2951438 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 2951438 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
   [junit4]   2> 2951438 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
   [junit4]   2> 2951438 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 2951438 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 2951468 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 2951468 DEBUG (Simulated OverseerAutoScalingTriggerThread) [    ] o.a.s.c.a.Ov

[...truncated too long message...]

etty-closer-7111-thread-11) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5b23f68e{/solr,null,UNAVAILABLE}
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-1) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-1) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 0
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 0
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:59545_solr
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:51737_solr
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:46836_solr
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:38172_solr
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:38825_solr
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:56216_solr
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:59700_solr
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:53879_solr
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:55145_solr
   [junit4]   2> 9426040 DEBUG (ScheduledTrigger-8574-thread-4) [n:127.0.0.1:56216_solr    ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:34726_solr
   [junit4]   2> 9426095 DEBUG (jetty-closer-7111-thread-1) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 9426098 INFO  (jetty-closer-7111-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@625b5b4a{/solr,null,UNAVAILABLE}
   [junit4]   2> 9426118 ERROR (SUITE-TriggerIntegrationTest-seed#[E7C582025295575F]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 9426118 INFO  (SUITE-TriggerIntegrationTest-seed#[E7C582025295575F]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:57629 57629
   [junit4]   2> 9426226 INFO  (Thread-4358) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:57629 57629
   [junit4]   2> 9426227 WARN  (Thread-4358) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	20	/solr/aliases.json
   [junit4]   2> 	19	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	20	/solr/clusterprops.json
   [junit4]   2> 	20	/solr/clusterstate.json
   [junit4]   2> 	5	/solr/autoscaling.json
   [junit4]   2> 	4	/solr/overseer_elect/election/99198192212246569-127.0.0.1:53879_solr-n_0000000012
   [junit4]   2> 	4	/solr/overseer_elect/election/99198192212246557-127.0.0.1:51737_solr-n_0000000008
   [junit4]   2> 	2	/solr/overseer_elect/election/99198192212246533-127.0.0.1:50896_solr-n_0000000001
   [junit4]   2> 	2	/solr/overseer_elect/election/99198192212246545-127.0.0.1:35594_solr-n_0000000004
   [junit4]   2> 	2	/solr/overseer_elect/election/99198192212246551-127.0.0.1:56216_solr-n_0000000006
   [junit4]   2> 	2	/solr/overseer_elect/election/99198192212246563-127.0.0.1:46836_solr-n_0000000010
   [junit4]   2> 	2	/solr/overseer_elect/election/99198192212246560-127.0.0.1:59545_solr-n_0000000009
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	20	/solr/live_nodes
   [junit4]   2> 	20	/solr/collections
   [junit4]   2> 	5	/solr/overseer/queue-work
   [junit4]   2> 	5	/solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 	5	/solr/overseer/queue
   [junit4]   2> 	5	/solr/overseer/collection-queue-work
   [junit4]   2> 	3	/solr/autoscaling/events/node_added_trigger
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.TriggerIntegrationTest_E7C582025295575F-001
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=23481, maxDocsPerChunk=729, blockSize=3), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=23481, blockSize=3)), sim=RandomSimilarity(queryNorm=true): {}, locale=fi, timezone=Africa/Niamey
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=216515272,total=504889344
   [junit4]   2> NOTE: All tests run in this JVM: [StatsReloadRaceTest, SpatialRPTFieldTypeTest, SolrSlf4jReporterTest, DistanceFunctionTest, TestTolerantSearch, DistributedSpellCheckComponentTest, HdfsChaosMonkeyNothingIsSafeTest, CdcrReplicationDistributedZkTest, RecoveryZkTest, RulesTest, SolrMetricReporterTest, TestPartialUpdateDeduplication, LoggingHandlerTest, IndexSchemaTest, TestBulkSchemaAPI, TestNRTOpen, TestLegacyFieldReuse, TestPullReplica, DistributedFacetPivotLongTailTest, ComputePlanActionTest, CopyFieldTest, AutoscalingHistoryHandlerTest, TestConfigSetImmutable, OverseerTest, SpellPossibilityIteratorTest, ZkNodePropsTest, ChaosMonkeyNothingIsSafeTest, RequiredFieldsTest, SolrGraphiteReporterTest, MigrateRouteKeyTest, TestManagedResource, TimeZoneUtilsTest, TestSolrCloudWithHadoopAuthPlugin, TestJsonFacetRefinement, CSVRequestHandlerTest, FieldAnalysisRequestHandlerTest, LeaderElectionContextKeyTest, TestRealTimeGet, TestUnifiedSolrHighlighter, TestIndexSearcher, DefaultValueUpdateProcessorTest, TestDocBasedVersionConstraints, DistributedFacetPivotWhiteBoxTest, TestIntervalFaceting, TestFieldCacheReopen, TestSolrCloudWithSecureImpersonation, CachingDirectoryFactoryTest, TestSchemaResource, SSLMigrationTest, SolrTestCaseJ4Test, AliasIntegrationTest, TestSolrCoreProperties, TestNumericTerms32, DistributedExpandComponentTest, TestPolicyCloud, BooleanFieldTest, SuggesterTest, DistribJoinFromCollectionTest, AlternateDirectoryTest, CloudMLTQParserTest, ReturnFieldsTest, BlockDirectoryTest, TestSkipOverseerOperations, TestValueSourceCache, PropertiesRequestHandlerTest, RegexBytesRefFilterTest, XsltUpdateRequestHandlerTest, HdfsNNFailoverTest, TestDFISimilarityFactory, PluginInfoTest, ClassificationUpdateProcessorTest, DirectSolrConnectionTest, TestPushWriter, QueryResultKeyTest, DirectSolrSpellCheckerTest, TestDistributedGrouping, TestCloudPivotFacet, TestSegmentSorting, TestSubQueryTransformer, TestScoreJoinQPNoScore, FileBasedSpellCheckerTest, LeaderInitiatedRecoveryOnCommitTest, TestExpandComponent, AtomicUpdatesTest, TestCorePropertiesReload, TestReRankQParserPlugin, SuggestComponentContextFilterQueryTest, TestSearchPerf, SharedFSAutoReplicaFailoverTest, SuggesterWFSTTest, TestStressLiveNodes, TestTrie, TestGroupingSearch, TestTolerantUpdateProcessorRandomCloud, UnloadDistributedZkTest, AsyncCallRequestStatusResponseTest, ZkStateWriterTest, SystemInfoHandlerTest, ReplicaListTransformerTest, TestCloudDeleteByQuery, BaseCdcrDistributedZkTest, HttpTriggerListenerTest, OverseerCollectionConfigSetProcessorTest, TestCloudSchemaless, RecoveryAfterSoftCommitTest, TriLevelCompositeIdRoutingTest, FullSolrCloudDistribCmdsTest, TestXmlQParser, DeleteStatusTest, XmlUpdateRequestHandlerTest, AddBlockUpdateTest, TestMacros, HdfsRestartWhileUpdatingTest, CdcrVersionReplicationTest, TestPointFields, TestCoreBackup, TestSolrDeletionPolicy2, SortByFunctionTest, HdfsLockFactoryTest, TestBinaryResponseWriter, SearchHandlerTest, TestImplicitCoreProperties, IndexBasedSpellCheckerTest, WrapperMergePolicyFactoryTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, LeaderElectionIntegrationTest, BasicZkTest, TestReplicationHandler, ZkControllerTest, TestStressReorder, TestJoin, TestReload, StatsComponentTest, BasicFunctionalityTest, TestLazyCores, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SuggesterFSTTest, CoreAdminHandlerTest, TestStressLucene, SpatialFilterTest, TestCSVLoader, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, CacheHeaderTest, DisMaxRequestHandlerTest, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, RequestHandlersTest, UpdateRequestProcessorFactoryTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, TestLFUCache, DistribCursorPagingTest, LeaderFailoverAfterPartitionTest, LeaderFailureAfterFreshStartTest, MoveReplicaTest, MultiThreadedOCPTest, ReplicationFactorTest, RollingRestartTest, SolrCloudExampleTest, TestClusterProperties, TestCollectionsAPIViaSolrCloudCluster, TestHdfsCloudBackupRestore, TestPrepRecovery, TestRequestForwarding, TestSizeLimitedDistributedMap, AutoAddReplicasPlanActionTest, ExecutePlanActionTest, NodeLostTriggerTest, TriggerIntegrationTest]
   [junit4] Completed [586/763 (8!)] on J2 in 189.90s, 13 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 48783 lines...]

[JENKINS] Lucene-Solr-Tests-7.x - Build # 285 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/285/

15 tests failed.
FAILED:  org.apache.solr.cloud.ConnectionManagerTest.testLikelyExpired

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:99FC22724F5227F7]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.junit.Assert.assertFalse(Assert.java:79)
	at org.apache.solr.cloud.ConnectionManagerTest.testLikelyExpired(ConnectionManagerTest.java:98)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.DistributedQueueTest.testDistributedQueueBlocking

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:F823F2532B7CBFBC]:0)
	at java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at org.apache.solr.cloud.DistributedQueueTest.testDistributedQueueBlocking(DistributedQueueTest.java:141)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.test

Error Message:
Error from server at http://127.0.0.1:35468/solr: KeeperErrorCode = Session expired for /overseer/collection-map-completed/mn-9ho2inv6dsz4yfz3c0eh0qjp0

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:35468/solr: KeeperErrorCode = Session expired for /overseer/collection-map-completed/mn-9ho2inv6dsz4yfz3c0eh0qjp0
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:35DDBFF0C1D26E30]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.MoveReplicaTest.test(MoveReplicaTest.java:151)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:35DDBFF0C1D26E30]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.MultiThreadedOCPTest.testFillWorkQueue(MultiThreadedOCPTest.java:112)
	at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.OverseerModifyCollectionTest.testModifyColl

Error Message:
KeeperErrorCode = Session expired for /collections/modifyColl

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/modifyColl
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:1411AA5F63E2C7CE]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:341)
	at org.apache.solr.cloud.OverseerModifyCollectionTest.getConfigNameFromZk(OverseerModifyCollectionTest.java:75)
	at org.apache.solr.cloud.OverseerModifyCollectionTest.testModifyColl(OverseerModifyCollectionTest.java:59)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader

Error Message:
Could not load collection from ZK: pull_replica_error_handling_test_cant_connect_to_leader

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: pull_replica_error_handling_test_cant_connect_to_leader
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1108)
	at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:647)
	at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:137)
	at org.apache.solr.cloud.TestPullReplicaErrorHandling.tearDown(TestPullReplicaErrorHandling.java:126)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/pull_replica_error_handling_test_cant_connect_to_leader/state.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1120)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1106)
	... 39 more


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue.testDistributedQueue

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8]:0)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testListeners

Error Message:
foo events: [CapturedEvent{timestamp=13781187167760510, stage=STARTED, actionName='null', event={   "id":"30f5e91c0dc812T9ho2inv6dsz4yfz3c0eh0qk0w",   "source":"node_added_trigger",   "eventTime":13781180428961810,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[13781180428961810],     "_enqueue_time_":13781183924653160,     "nodeNames":["127.0.0.1:10002_solr"]}}, context={}, config={   "trigger":"node_added_trigger",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "afterAction":[     "test",     "test1"],   "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",   "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=13781187885990510, stage=BEFORE_ACTION, actionName='test', event={   "id":"30f5e91c0dc812T9ho2inv6dsz4yfz3c0eh0qk0w",   "source":"node_added_trigger",   "eventTime":13781180428961810,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[13781180428961810],     "_enqueue_time_":13781183924653160,     "nodeNames":["127.0.0.1:10002_solr"]}}, context={properties.BEFORE_ACTION=[test], source=node_added_trigger}, config={   "trigger":"node_added_trigger",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "afterAction":[     "test",     "test1"],   "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",   "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=13781188843219960, stage=AFTER_ACTION, actionName='test', event={   "id":"30f5e91c0dc812T9ho2inv6dsz4yfz3c0eh0qk0w",   "source":"node_added_trigger",   "eventTime":13781180428961810,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[13781180428961810],     "_enqueue_time_":13781183924653160,     "nodeNames":["127.0.0.1:10002_solr"]}}, context={properties.BEFORE_ACTION=[test], source=node_added_trigger, properties.AFTER_ACTION=[test]}, config={   "trigger":"node_added_trigger",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "afterAction":[     "test",     "test1"],   "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",   "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=13781188844364360, stage=AFTER_ACTION, actionName='test1', event={   "id":"30f5e91c0dc812T9ho2inv6dsz4yfz3c0eh0qk0w",   "source":"node_added_trigger",   "eventTime":13781180428961810,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[13781180428961810],     "_enqueue_time_":13781183924653160,     "nodeNames":["127.0.0.1:10002_solr"]}}, context={properties.BEFORE_ACTION=[test, test1], source=node_added_trigger, properties.AFTER_ACTION=[test, test1]}, config={   "trigger":"node_added_trigger",   "stage":[     "STARTED",     "ABORTED",     "SUCCEEDED",     "FAILED"],   "afterAction":[     "test",     "test1"],   "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",   "beforeAction":"test"}, message='null'}] expected:<5> but was:<4>

Stack Trace:
java.lang.AssertionError: foo events: [CapturedEvent{timestamp=13781187167760510, stage=STARTED, actionName='null', event={
  "id":"30f5e91c0dc812T9ho2inv6dsz4yfz3c0eh0qk0w",
  "source":"node_added_trigger",
  "eventTime":13781180428961810,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[13781180428961810],
    "_enqueue_time_":13781183924653160,
    "nodeNames":["127.0.0.1:10002_solr"]}}, context={}, config={
  "trigger":"node_added_trigger",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "afterAction":[
    "test",
    "test1"],
  "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
  "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=13781187885990510, stage=BEFORE_ACTION, actionName='test', event={
  "id":"30f5e91c0dc812T9ho2inv6dsz4yfz3c0eh0qk0w",
  "source":"node_added_trigger",
  "eventTime":13781180428961810,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[13781180428961810],
    "_enqueue_time_":13781183924653160,
    "nodeNames":["127.0.0.1:10002_solr"]}}, context={properties.BEFORE_ACTION=[test], source=node_added_trigger}, config={
  "trigger":"node_added_trigger",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "afterAction":[
    "test",
    "test1"],
  "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
  "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=13781188843219960, stage=AFTER_ACTION, actionName='test', event={
  "id":"30f5e91c0dc812T9ho2inv6dsz4yfz3c0eh0qk0w",
  "source":"node_added_trigger",
  "eventTime":13781180428961810,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[13781180428961810],
    "_enqueue_time_":13781183924653160,
    "nodeNames":["127.0.0.1:10002_solr"]}}, context={properties.BEFORE_ACTION=[test], source=node_added_trigger, properties.AFTER_ACTION=[test]}, config={
  "trigger":"node_added_trigger",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "afterAction":[
    "test",
    "test1"],
  "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
  "beforeAction":"test"}, message='null'}, CapturedEvent{timestamp=13781188844364360, stage=AFTER_ACTION, actionName='test1', event={
  "id":"30f5e91c0dc812T9ho2inv6dsz4yfz3c0eh0qk0w",
  "source":"node_added_trigger",
  "eventTime":13781180428961810,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[13781180428961810],
    "_enqueue_time_":13781183924653160,
    "nodeNames":["127.0.0.1:10002_solr"]}}, context={properties.BEFORE_ACTION=[test, test1], source=node_added_trigger, properties.AFTER_ACTION=[test, test1]}, config={
  "trigger":"node_added_trigger",
  "stage":[
    "STARTED",
    "ABORTED",
    "SUCCEEDED",
    "FAILED"],
  "afterAction":[
    "test",
    "test1"],
  "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
  "beforeAction":"test"}, message='null'}] expected:<5> but was:<4>
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:5049173AD2429E11]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testListeners(TestTriggerIntegration.java:963)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState

Error Message:
The trigger did not fire at all

Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:BDBF34EA4623A7A2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testEventFromRestoredState(TestTriggerIntegration.java:673)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10008_solr should have been deleted

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10008_solr should have been deleted
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:A5330826611BCE27]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration(TestTriggerIntegration.java:842)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:E0C19EA3A0E8A587]:0)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1181)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeAddedTriggerRestoreState

Error Message:
Two TriggerAction instances should have been created by now

Stack Trace:
java.lang.AssertionError: Two TriggerAction instances should have been created by now
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:35B4095555EEE265]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeAddedTriggerRestoreState(TestTriggerIntegration.java:361)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testTriggerThrottling

Error Message:
Both triggers should have fired by now

Stack Trace:
java.lang.AssertionError: Both triggers should have fired by now
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:46AB280FBD84E05A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testTriggerThrottling(TestTriggerIntegration.java:171)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:37611_solr wasn't created

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:37611_solr wasn't created
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8:A5330826611BCE27]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration(TriggerIntegrationTest.java:932)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8]:0)




Build Log:
[...truncated 11873 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestPullReplicaErrorHandling
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BD89802A6F2E03C8-001/init-core-data-001
   [junit4]   2> 360385 WARN  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 360386 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 360387 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 360387 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 360387 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BD89802A6F2E03C8-001/tempDir-001
   [junit4]   2> 360388 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 360632 INFO  (Thread-142) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 360632 INFO  (Thread-142) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 360644 ERROR (Thread-142) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 360748 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:43268
   [junit4]   2> 360788 INFO  (zkConnectionManagerCallback-209-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 360825 INFO  (jetty-launcher-206-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 360825 INFO  (jetty-launcher-206-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 360826 INFO  (jetty-launcher-206-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 360827 INFO  (jetty-launcher-206-thread-4) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 360828 INFO  (jetty-launcher-206-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6e91b04d{/solr,null,AVAILABLE}
   [junit4]   2> 360828 INFO  (jetty-launcher-206-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7b198fd2{HTTP/1.1,[http/1.1]}{127.0.0.1:40234}
   [junit4]   2> 360828 INFO  (jetty-launcher-206-thread-3) [    ] o.e.j.s.Server Started @381431ms
   [junit4]   2> 360828 INFO  (jetty-launcher-206-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40234}
   [junit4]   2> 360828 ERROR (jetty-launcher-206-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 360829 INFO  (jetty-launcher-206-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 360829 INFO  (jetty-launcher-206-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 360829 INFO  (jetty-launcher-206-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 360829 INFO  (jetty-launcher-206-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-17T16:37:00.757Z
   [junit4]   2> 361248 INFO  (jetty-launcher-206-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@689a4769{/solr,null,AVAILABLE}
   [junit4]   2> 361248 INFO  (jetty-launcher-206-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7b77258a{HTTP/1.1,[http/1.1]}{127.0.0.1:50619}
   [junit4]   2> 361249 INFO  (jetty-launcher-206-thread-2) [    ] o.e.j.s.Server Started @381852ms
   [junit4]   2> 361249 INFO  (jetty-launcher-206-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50619}
   [junit4]   2> 361249 ERROR (jetty-launcher-206-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 361249 INFO  (jetty-launcher-206-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 361249 INFO  (jetty-launcher-206-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 361249 INFO  (jetty-launcher-206-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 361249 INFO  (jetty-launcher-206-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-17T16:37:01.177Z
   [junit4]   2> 361930 INFO  (jetty-launcher-206-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@21e4e398{/solr,null,AVAILABLE}
   [junit4]   2> 361930 INFO  (jetty-launcher-206-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2fd5c900{HTTP/1.1,[http/1.1]}{127.0.0.1:33737}
   [junit4]   2> 361930 INFO  (jetty-launcher-206-thread-1) [    ] o.e.j.s.Server Started @382533ms
   [junit4]   2> 361930 INFO  (jetty-launcher-206-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33737}
   [junit4]   2> 361931 ERROR (jetty-launcher-206-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 361931 INFO  (jetty-launcher-206-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 361931 INFO  (jetty-launcher-206-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 361931 INFO  (jetty-launcher-206-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 361931 INFO  (jetty-launcher-206-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-17T16:37:01.859Z
   [junit4]   2> 363322 INFO  (jetty-launcher-206-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6e8e0580{/solr,null,AVAILABLE}
   [junit4]   2> 363322 INFO  (jetty-launcher-206-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d24ef11{HTTP/1.1,[http/1.1]}{127.0.0.1:59510}
   [junit4]   2> 363322 INFO  (jetty-launcher-206-thread-4) [    ] o.e.j.s.Server Started @383925ms
   [junit4]   2> 363322 INFO  (jetty-launcher-206-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59510}
   [junit4]   2> 363323 ERROR (jetty-launcher-206-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 363323 INFO  (jetty-launcher-206-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 363323 INFO  (jetty-launcher-206-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 363323 INFO  (jetty-launcher-206-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 363323 INFO  (jetty-launcher-206-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-17T16:37:03.251Z
   [junit4]   2> 364218 INFO  (zkConnectionManagerCallback-213-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 364219 INFO  (zkConnectionManagerCallback-211-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 364238 INFO  (zkConnectionManagerCallback-217-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 364268 INFO  (zkConnectionManagerCallback-215-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 364269 INFO  (jetty-launcher-206-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 364269 INFO  (jetty-launcher-206-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 364326 INFO  (jetty-launcher-206-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43268/solr
   [junit4]   2> 364372 INFO  (jetty-launcher-206-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43268/solr
   [junit4]   2> 364577 INFO  (jetty-launcher-206-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 364584 INFO  (jetty-launcher-206-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 364585 INFO  (zkConnectionManagerCallback-221-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 364618 INFO  (jetty-launcher-206-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43268/solr
   [junit4]   2> 364668 INFO  (zkConnectionManagerCallback-225-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 364696 INFO  (jetty-launcher-206-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43268/solr
   [junit4]   2> 364727 INFO  (zkConnectionManagerCallback-235-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 364728 INFO  (zkConnectionManagerCallback-229-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 364834 INFO  (zkConnectionManagerCallback-239-thread-1-processing-n:127.0.0.1:59510_solr) [n:127.0.0.1:59510_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 364835 INFO  (zkConnectionManagerCallback-241-thread-1-processing-n:127.0.0.1:33737_solr) [n:127.0.0.1:33737_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 365059 INFO  (zkConnectionManagerCallback-233-thread-1-processing-n:127.0.0.1:50619_solr) [n:127.0.0.1:50619_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 365079 INFO  (zkConnectionManagerCallback-237-thread-1-processing-n:127.0.0.1:40234_solr) [n:127.0.0.1:40234_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 365185 INFO  (jetty-launcher-206-thread-2) [n:127.0.0.1:50619_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 365185 INFO  (jetty-launcher-206-thread-3) [n:127.0.0.1:40234_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 365186 INFO  (jetty-launcher-206-thread-3) [n:127.0.0.1:40234_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40234_solr
   [junit4]   2> 365186 INFO  (jetty-launcher-206-thread-2) [n:127.0.0.1:50619_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50619_solr
   [junit4]   2> 365256 INFO  (jetty-launcher-206-thread-3) [n:127.0.0.1:40234_solr    ] o.a.s.c.Overseer Overseer (id=99190611677282316-127.0.0.1:40234_solr-n_0000000000) starting
   [junit4]   2> 365745 INFO  (jetty-launcher-206-thread-4) [n:127.0.0.1:59510_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 365753 INFO  (jetty-launcher-206-thread-1) [n:127.0.0.1:33737_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 365773 INFO  (jetty-launcher-206-thread-4) [n:127.0.0.1:59510_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 365773 INFO  (jetty-launcher-206-thread-1) [n:127.0.0.1:33737_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 365775 INFO  (jetty-launcher-206-thread-4) [n:127.0.0.1:59510_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59510_solr
   [junit4]   2> 366056 INFO  (jetty-launcher-206-thread-2) [n:127.0.0.1:50619_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50619.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 366115 INFO  (jetty-launcher-206-thread-2) [n:127.0.0.1:50619_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50619.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 366156 INFO  (jetty-launcher-206-thread-2) [n:127.0.0.1:50619_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_50619.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 366157 INFO  (jetty-launcher-206-thread-2) [n:127.0.0.1:50619_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BD89802A6F2E03C8-001/tempDir-001/node2/.
   [junit4]   2> 366366 INFO  (jetty-launcher-206-thread-3) [n:127.0.0.1:40234_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40234_solr
   [junit4]   2> 366517 INFO  (jetty-launcher-206-thread-1) [n:127.0.0.1:33737_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33737_solr
   [junit4]   2> 368817 INFO  (zkCallback-238-thread-1-processing-n:127.0.0.1:59510_solr) [n:127.0.0.1:59510_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 368876 INFO  (zkCallback-236-thread-1-processing-n:127.0.0.1:40234_solr) [n:127.0.0.1:40234_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 368876 INFO  (zkCallback-232-thread-1-processing-n:127.0.0.1:50619_solr) [n:127.0.0.1:50619_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 369092 INFO  (zkCallback-240-thread-1-processing-n:127.0.0.1:33737_solr) [n:127.0.0.1:33737_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 369117 INFO  (zkCallback-232-thread-2-processing-n:127.0.0.1:50619_solr) [n:127.0.0.1:50619_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 369144 INFO  (jetty-launcher-206-thread-4) [n:127.0.0.1:59510_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59510.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 369269 INFO  (jetty-launcher-206-thread-4) [n:127.0.0.1:59510_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59510.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 369269 INFO  (jetty-launcher-206-thread-4) [n:127.0.0.1:59510_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59510.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 369270 INFO  (jetty-launcher-206-thread-4) [n:127.0.0.1:59510_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BD89802A6F2E03C8-001/tempDir-001/node4/.
   [junit4]   2> 370199 INFO  (jetty-launcher-206-thread-1) [n:127.0.0.1:33737_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33737.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 370251 INFO  (jetty-launcher-206-thread-3) [n:127.0.0.1:40234_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40234.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 370481 INFO  (jetty-launcher-206-thread-1) [n:127.0.0.1:33737_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33737.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 370481 INFO  (jetty-launcher-206-thread-1) [n:127.0.0.1:33737_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33737.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 370482 INFO  (jetty-launcher-206-thread-3) [n:127.0.0.1:40234_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40234.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 370482 INFO  (jetty-launcher-206-thread-3) [n:127.0.0.1:40234_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40234.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b5ace3e
   [junit4]   2> 370485 INFO  (jetty-launcher-206-thread-1) [n:127.0.0.1:33737_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BD89802A6F2E03C8-001/tempDir-001/node1/.
   [junit4]   2> 370485 INFO  (jetty-launcher-206-thread-3) [n:127.0.0.1:40234_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BD89802A6F2E03C8-001/tempDir-001/node3/.
   [junit4]   2> 372786 INFO  (zkConnectionManagerCallback-251-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 372905 INFO  (zkConnectionManagerCallback-255-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 373076 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 373084 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43268/solr ready
   [junit4]   2> 373148 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@7b77258a{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 373149 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1472580044
   [junit4]   2> 373196 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 373240 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4097a3d6: rootName = solr_50619, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@406f0c41
   [junit4]   2> 373280 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 373280 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1909eca9: rootName = solr_50619, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5b3da316
   [junit4]   2> 373306 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 373306 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2a147b9d: rootName = solr_50619, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@48a3a020
   [junit4]   2> 373318 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50619_solr
   [junit4]   2> 373320 INFO  (zkCallback-240-thread-1-processing-n:127.0.0.1:33737_solr) [n:127.0.0.1:33737_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 373320 INFO  (zkCallback-238-thread-1-processing-n:127.0.0.1:59510_solr) [n:127.0.0.1:59510_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 373320 INFO  (zkCallback-236-thread-2-processing-n:127.0.0.1:40234_solr) [n:127.0.0.1:40234_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 373322 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 373322 INFO  (zkCallback-232-thread-2-processing-n:127.0.0.1:50619_solr) [n:127.0.0.1:50619_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 373361 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 373364 INFO  (zkCallback-254-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
   [junit4]   2> 373373 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@689a4769{/solr,null,UNAVAILABLE}
   [junit4]   2> 373404 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 373405 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@56ff33d7{/solr,null,AVAILABLE}
   [junit4]   2> 373406 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2042e352{HTTP/1.1,[http/1.1]}{127.0.0.1:33574}
   [junit4]   2> 373406 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.e.j.s.Server Started @394009ms
   [junit4]   2> 373417 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50323}
   [junit4]   2> 373417 ERROR (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 373417 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 373417 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 373417 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 373417 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-17T16:37:13.345Z
   [junit4]   2> 373536 INFO  (zkConnectionManagerCallback-257-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 373544 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 373668 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43268/solr
   [junit4]   2> 373733 INFO  (zkConnectionManagerCallback-261-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 373864 INFO  (zkConnectionManagerCallback-263-thread-1-processing-n:127.0.0.1:50323_solr) [n:127.0.0.1:50323_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 373868 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [n:127.0.0.1:50323_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 373870 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89802A6F2E03C8]-worker) [n:127.0.0.1:50323_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 373877 INFO  (SUITE-TestPullReplicaErrorHandling-seed#[BD89

[...truncated too long message...]

P-TestGenericDistributedQueue]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueue(TestSimDistributedQueue.java:73)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> ??? 18, 2017 5:46:17 ? com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=11294, name=TEST-TestGenericDistributedQueue.testDistributedQueue-seed#[BD89802A6F2E03C8], state=TIMED_WAITING, group=TGRP-TestGenericDistributedQueue]
   [junit4]   2> 11714864 INFO  (TEST-TestGenericDistributedQueue.testDistributedQueue-seed#[BD89802A6F2E03C8]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueue
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestGenericDistributedQueue_BD89802A6F2E03C8-001
   [junit4]   2> Dec 17, 2017 7:46:17 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-LY, timezone=Pacific/Guam
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=120012504,total=530579456
   [junit4]   2> NOTE: All tests run in this JVM: [NodeLostTriggerTest, TestCollationField, TestFaceting, TestAnalyzedSuggestions, BitVectorTest, DistributedIntervalFacetingTest, TestSmileRequest, TestSQLHandler, TestRuleBasedAuthorizationPlugin, TestLegacyField, TestNoOpRegenerator, HLLSerializationTest, TestTriggerIntegration, TestRandomFlRTGCloud, TestSolrCloudWithKerberosAlt, TestDeleteCollectionOnDownNodes, MBeansHandlerTest, TestJavabinTupleStreamParser, PKIAuthenticationIntegrationTest, PingRequestHandlerTest, DistribDocExpirationUpdateProcessorTest, ZkStateReaderTest, AssignTest, PathHierarchyTokenizerFactoryTest, TestSolrCLIRunExample, TestBM25SimilarityFactory, TestLeaderElectionZkExpiry, DocExpirationUpdateProcessorFactoryTest, OutputWriterTest, TestConfigSetsAPIExclusivity, TestStressRecovery, TestHalfAndHalfDocValues, CloudExitableDirectoryReaderTest, ClusterStateUpdateTest, TestMaxScoreQueryParser, TestPivotHelperCode, DirectUpdateHandlerTest, AutoScalingHandlerTest, TestLegacyFieldCache, CdcrRequestHandlerTest, TestLegacyTerms, LegacyCloudClusterPropTest, TestRemoteStreaming, TestChildDocTransformer, SchemaVersionSpecificBehaviorTest, DeleteShardTest, LukeRequestHandlerTest, TestSolrFieldCacheBean, TestManagedStopFilterFactory, HighlighterMaxOffsetTest, V2StandaloneTest, SOLR749Test, TestOmitPositions, TestRangeQuery, SolrMetricManagerTest, TestDocSet, RequestHandlersTest, HdfsDirectoryFactoryTest, BinaryUpdateRequestHandlerTest, DistributedFacetSimpleRefinementLongTailTest, TestPrepRecovery, TestPerFieldSimilarityWithDefaultOverride, CreateCollectionCleanupTest, RegexBoostProcessorTest, SpellingQueryConverterTest, TestSimpleTrackingShardHandler, TestQuerySenderNoQuery, TestSolrXml, TestMinMaxOnMultiValuedField, TestSchemaVersionResource, TestMaxTokenLenTokenizer, SolrShardReporterTest, DistributedTermsComponentTest, TestFastWriter, PolyFieldTest, ConcurrentDeleteAndCreateCollectionTest, ZkCLITest, TestTrieFacet, InfixSuggestersTest, HdfsBasicDistributedZkTest, ClassificationUpdateProcessorIntegrationTest, TestMacroExpander, TestExclusionRuleCollectionAccess, TestSchemalessBufferedUpdates, TestComplexPhraseLeadingWildcard, TestImpersonationWithHadoopAuth, SynonymTokenizerTest, TestAtomicUpdateErrorCases, LargeFieldTest, TestManagedSchemaThreadSafety, MetricsHandlerTest, TestReplicationHandlerBackup, HighlighterConfigTest, MoveReplicaHDFSFailoverTest, OverseerRolesTest, DistributedQueryComponentOptimizationTest, NumericFieldsTest, TestUninvertingReader, TestDynamicLoading, TestFieldResource, TestCursorMarkWithoutUniqueKey, DebugComponentTest, DistributedDebugComponentTest, MoreLikeThisHandlerTest, TestSolrQueryParser, TestExportWriter, SampleTest, TestGenericDistributedQueue]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGenericDistributedQueue -Dtests.seed=BD89802A6F2E03C8 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar-LY -Dtests.timezone=Pacific/Guam -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | TestGenericDistributedQueue (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BD89802A6F2E03C8]:0)
   [junit4] Completed [690/763 (9!)] on J1 in 7221.33s, 3 tests, 2 errors <<< FAILURES!

[...truncated 48360 lines...]

[JENKINS] Lucene-Solr-Tests-7.x - Build # 284 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/284/

6 tests failed.
FAILED:  org.apache.lucene.analysis.opennlp.TestOpenNLPPOSFilterFactory.testNoBreak

Error Message:
term 0 expected:<[Noo]> but was:<[peeriood]>

Stack Trace:
org.junit.ComparisonFailure: term 0 expected:<[Noo]> but was:<[peeriood]>
	at __randomizedtesting.SeedInfo.seed([944F0A574A20C2E8:71AAAAEEE56F1F91]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:201)
	at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:325)
	at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:329)
	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:865)
	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:727)
	at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:390)
	at org.apache.lucene.analysis.opennlp.TestOpenNLPPOSFilterFactory.testNoBreak(TestOpenNLPPOSFilterFactory.java:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddTooManyReplicas

Error Message:
Could not load collection from ZK: TooManyReplicasInSeveralFlavors

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: TooManyReplicasInSeveralFlavors
	at __randomizedtesting.SeedInfo.seed([7B5E06FB9A906FD4:F60C7223D4765065]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1108)
	at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:647)
	at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:137)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:117)
	at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
	at org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddTooManyReplicas(CollectionTooManyReplicasTest.java:91)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/TooManyReplicasInSeveralFlavors/state.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1120)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1106)
	... 44 more


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
	at __randomizedtesting.SeedInfo.seed([7B5E06FB9A906FD4:B2EB445593F7A921]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:707)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:35320_solr wasn't created

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:35320_solr wasn't created
	at __randomizedtesting.SeedInfo.seed([7B5E06FB9A906FD4:63E48EF794A5A23B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration(TriggerIntegrationTest.java:932)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue.testDistributedQueueBlocking

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
	at __randomizedtesting.SeedInfo.seed([7B5E06FB9A906FD4:3EF47482DEC2D3A0]:0)
	at java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueueBlocking(TestSimDistributedQueue.java:116)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue:     1) Thread[id=6352, name=sdqtest--2548-thread-1, state=TIMED_WAITING, group=TGRP-TestGenericDistributedQueue]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$1(TestSimDistributedQueue.java:107)         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$489/327199925.call(Unknown Source)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$113/1886776189.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue: 
   1) Thread[id=6352, name=sdqtest--2548-thread-1, state=TIMED_WAITING, group=TGRP-TestGenericDistributedQueue]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
        at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
        at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$1(TestSimDistributedQueue.java:107)
        at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$489/327199925.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$113/1886776189.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([7B5E06FB9A906FD4]:0)




Build Log:
[...truncated 3941 lines...]
   [junit4] Suite: org.apache.lucene.analysis.opennlp.TestOpenNLPPOSFilterFactory
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestOpenNLPPOSFilterFactory -Dtests.method=testNoBreak -Dtests.seed=944F0A574A20C2E8 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar-KW -Dtests.timezone=Eire -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.10s J2 | TestOpenNLPPOSFilterFactory.testNoBreak <<<
   [junit4]    > Throwable #1: org.junit.ComparisonFailure: term 0 expected:<[Noo]> but was:<[peeriood]>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([944F0A574A20C2E8:71AAAAEEE56F1F91]:0)
   [junit4]    > 	at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:201)
   [junit4]    > 	at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:325)
   [junit4]    > 	at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:329)
   [junit4]    > 	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:865)
   [junit4]    > 	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:727)
   [junit4]    > 	at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:390)
   [junit4]    > 	at org.apache.lucene.analysis.opennlp.TestOpenNLPPOSFilterFactory.testNoBreak(TestOpenNLPPOSFilterFactory.java:92)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1382, maxMBSortInHeap=5.683165736407708, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-KW, timezone=Eire
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=212063392,total=253231104
   [junit4]   2> NOTE: All tests run in this JVM: [TestOpenNLPPOSFilterFactory]
   [junit4] Completed [2/5 (1!)] on J2 in 8.73s, 3 tests, 1 failure <<< FAILURES!

[...truncated 8009 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
   [junit4]   2> 572286 INFO  (SUITE-TriggerIntegrationTest-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7B5E06FB9A906FD4-001/init-core-data-001
   [junit4]   2> 572287 WARN  (SUITE-TriggerIntegrationTest-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 572303 INFO  (SUITE-TriggerIntegrationTest-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 572305 INFO  (SUITE-TriggerIntegrationTest-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 572306 INFO  (SUITE-TriggerIntegrationTest-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7B5E06FB9A906FD4-001/tempDir-001
   [junit4]   2> 572306 INFO  (SUITE-TriggerIntegrationTest-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 572323 INFO  (Thread-2074) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 572323 INFO  (Thread-2074) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 572346 ERROR (Thread-2074) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 572413 INFO  (SUITE-TriggerIntegrationTest-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:41527
   [junit4]   2> 572435 INFO  (zkConnectionManagerCallback-344-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 572440 INFO  (jetty-launcher-341-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 572445 INFO  (jetty-launcher-341-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 572446 INFO  (jetty-launcher-341-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2d3068a2{/solr,null,AVAILABLE}
   [junit4]   2> 572447 INFO  (jetty-launcher-341-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@76972654{SSL,[ssl, http/1.1]}{127.0.0.1:43568}
   [junit4]   2> 572447 INFO  (jetty-launcher-341-thread-1) [    ] o.e.j.s.Server Started @585981ms
   [junit4]   2> 572447 INFO  (jetty-launcher-341-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43568}
   [junit4]   2> 572447 ERROR (jetty-launcher-341-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 572447 INFO  (jetty-launcher-341-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 572447 INFO  (jetty-launcher-341-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 572447 INFO  (jetty-launcher-341-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 572447 INFO  (jetty-launcher-341-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-15T22:45:17.611Z
   [junit4]   2> 572490 INFO  (jetty-launcher-341-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4de1bcc2{/solr,null,AVAILABLE}
   [junit4]   2> 572490 INFO  (jetty-launcher-341-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4f90cc9c{SSL,[ssl, http/1.1]}{127.0.0.1:33764}
   [junit4]   2> 572490 INFO  (jetty-launcher-341-thread-2) [    ] o.e.j.s.Server Started @586024ms
   [junit4]   2> 572490 INFO  (jetty-launcher-341-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33764}
   [junit4]   2> 572490 ERROR (jetty-launcher-341-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 572491 INFO  (jetty-launcher-341-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 572491 INFO  (jetty-launcher-341-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 572491 INFO  (jetty-launcher-341-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 572491 INFO  (jetty-launcher-341-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-15T22:45:17.655Z
   [junit4]   2> 572512 INFO  (zkConnectionManagerCallback-346-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 572534 INFO  (zkConnectionManagerCallback-348-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 572535 INFO  (jetty-launcher-341-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 572535 INFO  (jetty-launcher-341-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 572612 INFO  (jetty-launcher-341-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41527/solr
   [junit4]   2> 572623 INFO  (jetty-launcher-341-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41527/solr
   [junit4]   2> 572638 INFO  (zkConnectionManagerCallback-352-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 572661 INFO  (zkConnectionManagerCallback-356-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 572662 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1605c5a6a300003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 572664 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1605c5a6a300004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 572682 INFO  (zkConnectionManagerCallback-358-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 572682 INFO  (zkConnectionManagerCallback-360-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 572819 INFO  (jetty-launcher-341-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 572821 INFO  (jetty-launcher-341-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43568_solr
   [junit4]   2> 572822 INFO  (jetty-launcher-341-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.c.Overseer Overseer (id=99180735200493573-127.0.0.1:43568_solr-n_0000000000) starting
   [junit4]   2> 572854 INFO  (jetty-launcher-341-thread-2) [n:127.0.0.1:33764_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 572855 INFO  (jetty-launcher-341-thread-2) [n:127.0.0.1:33764_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33764_solr
   [junit4]   2> 572936 INFO  (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 572937 INFO  (zkCallback-357-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 572949 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 572950 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 572951 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:33764_solr]
   [junit4]   2> 572951 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 572951 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 572951 INFO  (jetty-launcher-341-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43568_solr
   [junit4]   2> 572952 INFO  (zkCallback-357-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 572957 INFO  (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 572985 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 573018 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 573018 DEBUG (ScheduledTrigger-602-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 573024 INFO  (jetty-launcher-341-thread-2) [n:127.0.0.1:33764_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33764.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 573033 INFO  (jetty-launcher-341-thread-2) [n:127.0.0.1:33764_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33764.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 573036 INFO  (jetty-launcher-341-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43568.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 573042 INFO  (jetty-launcher-341-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43568.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 573043 INFO  (jetty-launcher-341-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43568.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 573044 INFO  (jetty-launcher-341-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7B5E06FB9A906FD4-001/tempDir-001/node1/.
   [junit4]   2> 573044 INFO  (jetty-launcher-341-thread-2) [n:127.0.0.1:33764_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33764.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 573045 INFO  (jetty-launcher-341-thread-2) [n:127.0.0.1:33764_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7B5E06FB9A906FD4-001/tempDir-001/node2/.
   [junit4]   2> 573126 INFO  (zkConnectionManagerCallback-366-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 573157 INFO  (zkConnectionManagerCallback-370-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 573161 INFO  (SUITE-TriggerIntegrationTest-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 573163 INFO  (SUITE-TriggerIntegrationTest-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41527/solr ready
   [junit4] IGNOR/A 0.01s J2 | TriggerIntegrationTest.testSearchRate
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/SOLR-11714))
   [junit4]   2> 573189 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
   [junit4]   2> 573189 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
   [junit4]   2> 573189 DEBUG (zkCallback-357-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
   [junit4]   2> 573189 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
   [junit4]   2> 573195 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 573195 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 573196 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
   [junit4]   2> 573237 INFO  (qtp1516577509-3316) [n:127.0.0.1:33764_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 573287 INFO  (qtp1516577509-3316) [n:127.0.0.1:33764_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=49
   [junit4]   2> 573346 INFO  (qtp1516577509-3318) [n:127.0.0.1:33764_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 573366 INFO  (qtp2063385974-3309) [n:127.0.0.1:43568_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 573376 DEBUG (qtp2063385974-3311) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 573377 INFO  (qtp2063385974-3311) [n:127.0.0.1:43568_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=36
   [junit4]   2> 573377 DEBUG (zkCallback-357-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
   [junit4]   2> 573383 INFO  (zkCallback-357-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventQueueAction instantiated
   [junit4]   2> 573401 DEBUG (zkCallback-357-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:33764_solr, 127.0.0.1:43568_solr]
   [junit4]   2> 573401 DEBUG (zkCallback-357-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventQueueAction}], enabled=true}
   [junit4]   2> 573401 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
   [junit4]   2> 573402 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 573404 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 573434 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@42712720{/solr,null,AVAILABLE}
   [junit4]   2> 573435 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@410f5983{SSL,[ssl, http/1.1]}{127.0.0.1:42094}
   [junit4]   2> 573435 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.Server Started @586969ms
   [junit4]   2> 573435 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42094}
   [junit4]   2> 573435 ERROR (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 573444 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 573444 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 573444 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 573445 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-15T22:45:18.608Z
   [junit4]   2> 573437 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 573446 DEBUG (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
   [junit4]   2> 573445 DEBUG (ScheduledTrigger-602-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 573462 DEBUG (ScheduledTrigger-602-thread-1) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 573463 INFO  (zkConnectionManagerCallback-372-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 573478 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 573508 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41527/solr
   [junit4]   2> 573515 INFO  (zkConnectionManagerCallback-376-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 573523 INFO  (zkConnectionManagerCallback-378-thread-1-processing-n:127.0.0.1:42094_solr) [n:127.0.0.1:42094_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 573528 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:42094_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 573530 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:42094_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 573531 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:42094_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42094_solr
   [junit4]   2> 573533 INFO  (zkCallback-357-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 573533 INFO  (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 573534 INFO  (zkCallback-369-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 573569 INFO  (zkCallback-377-thread-1-processing-n:127.0.0.1:42094_solr) [n:127.0.0.1:42094_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 573605 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:42094_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42094.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 573614 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:42094_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42094.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 573619 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:42094_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42094.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 573620 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:42094_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7B5E06FB9A906FD4-001/tempDir-001/node3/.
   [junit4]   2> 574464 DEBUG (ScheduledTrigger-602-thread-2) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 574464 DEBUG (ScheduledTrigger-602-thread-2) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 574464 DEBUG (ScheduledTrigger-602-thread-2) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:42094_solr at time 13487228907252723
   [junit4]   2> 575271 INFO  (OverseerCollectionConfigSetProcessor-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 575490 DEBUG (ScheduledTrigger-602-thread-2) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 575490 DEBUG (ScheduledTrigger-602-thread-2) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 575490 DEBUG (ScheduledTrigger-602-thread-2) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:42094_solr] added at times [13487228907252723], now=13487229932830649
   [junit4]   2> 575490 DEBUG (ScheduledTrigger-602-thread-2) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"2fea9030056bf3T45lzyx6to8jpxgaejcx5c2r1",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":13487228907252723,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[13487228907252723],
   [junit4]   2>     "nodeNames":["127.0.0.1:42094_solr"]}}
   [junit4]   2> 575491 INFO  (ScheduledTrigger-602-thread-2) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 575492 DEBUG (AutoscalingActionExecutor-603-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"2fea9030056bf3T45lzyx6to8jpxgaejcx5c2r1",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":13487228907252723,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[13487228907252723],
   [junit4]   2>     "_enqueue_time_":13487229933245274,
   [junit4]   2>     "nodeNames":["127.0.0.1:42094_solr"]}}
   [junit4]   2> 575493 INFO  (AutoscalingActionExecutor-603-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.TriggerIntegrationTest -- event: {
   [junit4]   2>   "id":"2fea9030056bf3T45lzyx6to8jpxgaejcx5c2r1",
   [junit4]   2>   "source":"node_added_trigger1",
   [junit4]   2>   "eventTime":13487228907252723,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[13487228907252723],
   [junit4]   2>     "_enqueue_time_":13487229933245274,
   [junit4]   2>     "nodeNames":["127.0.0.1:42094_solr"]}}
   [junit4]   2> 575495 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@76972654{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 575498 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1820587091
   [junit4]   2> 575498 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 575509 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@69bfb02e: rootName = solr_43568, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@3b558355
   [junit4]   2> 575513 DEBUG (AutoscalingActionExecutor-603-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 21 ms for event id=2fea9030056bf3T45lzyx6to8jpxgaejcx5c2r1
   [junit4]   2> 575515 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 575515 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1d272afb: rootName = solr_43568, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@67e59e4d
   [junit4]   2> 575520 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 575520 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@34cb293: rootName = solr_43568, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@262a70df
   [junit4]   2> 575535 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:43568_solr
   [junit4]   2> 575536 INFO  (zkCallback-357-thread-1-processing-n:127.0.0.1:43568_solr) [n:127.0.0.1:43568_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 575537 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 575539 INFO  (zkCallback-377-thread-1-processing-n:127.0.0.1:42094_solr) [n:127.0.0.1:42094_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 575539 INFO  (zkCallback-369-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 575539 INFO  (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 575541 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 575541 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.Overseer Overseer (id=99180735200493573-127.0.0.1:43568_solr-n_0000000000) closing
   [junit4]   2> 575541 INFO  (OverseerStateUpdate-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:43568_solr
   [junit4]   2> 575543 WARN  (OverseerAutoScalingTriggerThread-99180735200493573-127.0.0.1:43568_solr-n_0000000000) [n:127.0.0.1:43568_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 575543 DEBUG (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 575545 INFO  (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33764_solr
   [junit4]   2> 575546 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2d3068a2{/solr,null,UNAVAILABLE}
   [junit4]   2> 575555 INFO  (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.Overseer Overseer (id=99180735200493574-127.0.0.1:33764_solr-n_0000000001) starting
   [junit4]   2> 575562 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 575563 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
   [junit4]   2> 575563 INFO  (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.TriggerIntegrationTest TestEventQueueAction instantiated
   [junit4]   2> 575563 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:33764_solr, 127.0.0.1:42094_solr]
   [junit4]   2> 575563 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventQueueAction}], enabled=true}
   [junit4]   2> 575564 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:33764_solr, 127.0.0.1:42094_solr]
   [junit4]   2> 575564 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion -1
   [junit4]   2> 575564 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
   [junit4]   2> 575565 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 575566 DEBUG (ScheduledTrigger-617-thread-1) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 575566 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
   [junit4]   2> 575567 DEBUG (ScheduledTrigger-617-thread-2) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 575567 DEBUG (ScheduledTrigger-617-thread-2) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 576566 DEBUG (ScheduledTrigger-617-thread-1) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 576568 DEBUG (ScheduledTrigger-617-thread-1) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 576568 DEBUG (ScheduledTrigger-617-thread-1) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 577568 DEBUG (ScheduledTrigger-617-thread-2) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 577568 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 577568 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 578569 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 578569 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 578569 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 579570 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 579570 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 579570 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 580574 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 580574 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 580574 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 581574 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 581574 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 581575 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 582579 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 582579 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 582579 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 583563 INFO  (TEST-TriggerIntegrationTest.testEventQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEventQueue
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TriggerIntegrationTest -Dtests.method=testEventQueue -Dtests.seed=7B5E06FB9A906FD4 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=en-IN -Dtests.timezone=Asia/Yakutsk -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 10.4s J2 | TriggerIntegrationTest.testEventQueue <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: action wasn't interrupted
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7B5E06FB9A906FD4:B2EB445593F7A921]:0)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventQueue(TriggerIntegrationTest.java:707)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 583579 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
   [junit4]   2> 583579 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
   [junit4]   2> 583579 DEBUG (ScheduledTrigger-617-thread-4) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 583629 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
   [junit4]   2> 583642 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
   [junit4]   2> 583653 DEBUG (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
   [junit4]   2> 583653 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
   [junit4]   2> 583689 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 583689 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 583689 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
   [junit4]   2> 583811 INFO  (qtp1516577509-3322) [n:127.0.0.1:33764_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 583832 INFO  (qtp1630285579-3374) [n:127.0.0.1:42094_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 583833 DEBUG (qtp1630285579-3375) [n:127.0.0.1:42094_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 583843 DEBUG (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
   [junit4]   2> 583864 INFO  (qtp1630285579-3375) [n:127.0.0.1:42094_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=82
   [junit4]   2> 583865 DEBUG (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:33764_solr, 127.0.0.1:42094_solr]
   [junit4]   2> 583865 DEBUG (zkCallback-359-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 583865 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
   [junit4]   2> 583866 INFO  (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 583880 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 583880 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
   [junit4]   2> 583940 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 583966 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 583966 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
   [junit4]   2> 583967 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7609514{/solr,null,AVAILABLE}
   [junit4]   2> 583967 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@a2b8960{SSL,[ssl, http/1.1]}{127.0.0.1:33787}
   [junit4]   2> 583967 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.Server Started @597502ms
   [junit4]   2> 583967 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33787}
   [junit4]   2> 583968 ERROR (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 583970 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 583970 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 583970 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 583970 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-15T22:45:29.134Z
   [junit4]   2> 584083 INFO  (zkConnectionManagerCallback-382-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 584221 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 584311 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41527/solr
   [junit4]   2> 584355 INFO  (zkConnectionManagerCallback-386-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 584397 INFO  (zkConnectionManagerCallback-388-thread-1-processing-n:127.0.0.1:33787_solr) [n:127.0.0.1:33787_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 584419 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:33787_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 584433 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:33787_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 584435 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:33787_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33787_solr
   [junit4]   2> 584449 INFO  (zkCallback-377-thread-1-processing-n:127.0.0.1:42094_solr) [n:127.0.0.1:42094_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 584473 INFO  (zkCallback-359-thread-3-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 584489 INFO  (zkCallback-369-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 584494 INFO  (zkCallback-387-thread-1-processing-n:127.0.0.1:33787_solr) [n:127.0.0.1:33787_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 584737 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:33787_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33787.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 584970 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:33787_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33787.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 584970 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:33787_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33787.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@15d892db
   [junit4]   2> 584971 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [n:127.0.0.1:33787_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7B5E06FB9A906FD4-001/tempDir-001/node4/.
   [junit4]   2> 584979 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 584979 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 584979 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:33787_solr at time 13487239422113232
   [junit4]   2> 586010 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 586010 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 586010 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:33787_solr] added at times [13487239422113232], now=13487240453189487
   [junit4]   2> 586010 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
   [junit4]   2>   "id":"2fea92a2c175d0T45lzyx6to8jpxgaejcx5c2r3",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":13487239422113232,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[13487239422113232],
   [junit4]   2>     "nodeNames":["127.0.0.1:33787_solr"]}}
   [junit4]   2> 586044 INFO  (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
   [junit4]   2> 586064 DEBUG (AutoscalingActionExecutor-618-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
   [junit4]   2>   "id":"2fea92a2c175d0T45lzyx6to8jpxgaejcx5c2r3",
   [junit4]   2>   "source":"node_added_trigger",
   [junit4]   2>   "eventTime":13487239422113232,
   [junit4]   2>   "eventType":"NODEADDED",
   [junit4]   2>   "properties":{
   [junit4]   2>     "eventTimes":[13487239422113232],
   [junit4]   2>     "_enqueue_time_":13487240453395964,
   [junit4]   2>     "nodeNames":["127.0.0.1:33787_solr"]}}
   [junit4]   2> 586065 DEBUG (AutoscalingActionExecutor-618-thread-1-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.ScheduledTriggers -- processing took 1 ms for event id=2fea92a2c175d0T45lzyx6to8jpxgaejcx5c2r3
   [junit4]   2> 586115 INFO  (qtp1516577509-3318) [n:127.0.0.1:33764_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 586298 INFO  (qtp2067371865-3414) [n:127.0.0.1:33787_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 586314 INFO  (qtp1630285579-3372) [n:127.0.0.1:42094_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 586314 DEBUG (qtp1516577509-3320) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 586324 INFO  (qtp1516577509-3320) [n:127.0.0.1:33764_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=257
   [junit4]   2> 586340 DEBUG (zkCallback-359-thread-3-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
   [junit4]   2> 586360 DEBUG (zkCallback-359-thread-3-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:33764_solr, 127.0.0.1:42094_solr, 127.0.0.1:33787_solr]
   [junit4]   2> 586360 DEBUG (zkCallback-359-thread-3-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 586361 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
   [junit4]   2> 586397 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 586434 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
   [junit4]   2> 587146 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 587146 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 588162 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 588162 DEBUG (ScheduledTrigger-617-thread-3) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 588345 INFO  (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
   [junit4]   2> 588413 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
   [junit4]   2> 588431 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 8
   [junit4]   2> 588434 DEBUG (zkCallback-359-thread-3-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
   [junit4]   2> 588435 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
   [junit4]   2> 588456 INFO  (qtp1516577509-3318) [n:127.0.0.1:33764_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 588471 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 588472 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 588472 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
   [junit4]   2> 588473 INFO  (qtp2067371865-3415) [n:127.0.0.1:33787_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 588474 INFO  (qtp1630285579-3372) [n:127.0.0.1:42094_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CORE.coreName&wt=javabin&version=2&group=solr.core} status=0 QTime=0
   [junit4]   2> 588493 DEBUG (qtp1516577509-3316) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
   [junit4]   2> 588494 DEBUG (zkCallback-359-thread-3-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
   [junit4]   2> 588494 DEBUG (zkCallback-359-thread-3-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:33764_solr, 127.0.0.1:42094_solr, 127.0.0.1:33787_solr]
   [junit4]   2> 588494 DEBUG (zkCallback-359-thread-3-processing-n:127.0.0.1:33764_solr) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
   [junit4]   2> 588494 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
   [junit4]   2> 588495 INFO  (qtp1516577509-3316) [n:127.0.0.1:33764_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=60
   [junit4]   2> 588508 INFO  (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
   [junit4]   2> 588508 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
   [junit4]   2> 588508 DEBUG (OverseerAutoScalingTriggerThread-99180735200493574-127.0.0.1:33764_solr-n_0000000001) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
   [junit4]   2> 588509 DEBUG (ScheduledTrigger-617-thread-1) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
   [junit4]   2> 588509 DEBUG (ScheduledTrigger-617-thread-1) [n:127.0.0.1:33764_solr    ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
   [junit4]   2> 588529 INFO  (qtp1516577509-3322) [n:127.0.0.1:33764_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 588535 INFO  (qtp1516577509-3322) [n:127.0.0.1:33764_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=6
   [junit4]   2> 588573 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 588576 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@cd5608f{/solr,null,AVAILABLE}
   [junit4]   2> 588576 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@129ab8ab{SSL,[ssl, http/1.1]}{127.0.0.1:36088}
   [junit4]   2> 588576 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.e.j.s.Server Started @602111ms
   [junit4]   2> 588576 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36088}
   [junit4]   2> 588628 ERROR (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 588628 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 588628 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 588628 INFO  (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null

[...truncated too long message...]

Test, AnalyticsMergeStrategyTest, RollingRestartTest, MetricsHandlerTest, TestDocSet, CollectionTooManyReplicasTest]
   [junit4] Completed [80/763 (2!)] on J1 in 59.51s, 3 tests, 1 error <<< FAILURES!

[...truncated 85 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue
   [junit4]   2> 1227896 INFO  (SUITE-TestGenericDistributedQueue-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestGenericDistributedQueue_7B5E06FB9A906FD4-001/init-core-data-001
   [junit4]   2> 1227897 INFO  (SUITE-TestGenericDistributedQueue-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1227898 INFO  (SUITE-TestGenericDistributedQueue-seed#[7B5E06FB9A906FD4]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1228238 INFO  (TEST-TestGenericDistributedQueue.testLocallyOffer-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLocallyOffer
   [junit4]   2> 1228790 INFO  (TEST-TestGenericDistributedQueue.testLocallyOffer-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testLocallyOffer
   [junit4]   2> 1228841 INFO  (TEST-TestGenericDistributedQueue.testPeekElements-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPeekElements
   [junit4]   2> 1230401 INFO  (TEST-TestGenericDistributedQueue.testPeekElements-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testPeekElements
   [junit4]   2> 1230455 INFO  (TEST-TestGenericDistributedQueue.testDistributedQueueBlocking-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueueBlocking
   [junit4]   2> 1233599 INFO  (TEST-TestGenericDistributedQueue.testDistributedQueueBlocking-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueueBlocking
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGenericDistributedQueue -Dtests.method=testDistributedQueueBlocking -Dtests.seed=7B5E06FB9A906FD4 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=tr -Dtests.timezone=US/Aleutian -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   3.20s J1 | TestGenericDistributedQueue.testDistributedQueueBlocking <<<
   [junit4]    > Throwable #1: java.util.concurrent.TimeoutException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7B5E06FB9A906FD4:3EF47482DEC2D3A0]:0)
   [junit4]    > 	at java.util.concurrent.FutureTask.get(FutureTask.java:205)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueueBlocking(TestSimDistributedQueue.java:116)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1233659 INFO  (TEST-TestGenericDistributedQueue.testDistributedQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueue
   [junit4]   2> 1235731 INFO  (TEST-TestGenericDistributedQueue.testDistributedQueue-seed#[7B5E06FB9A906FD4]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueue
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestGenericDistributedQueue_7B5E06FB9A906FD4-001
   [junit4]   2> Dec 15, 2017 10:56:20 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Dec 15, 2017 10:56:41 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue: 
   [junit4]   2>    1) Thread[id=6352, name=sdqtest--2548-thread-1, state=TIMED_WAITING, group=TGRP-TestGenericDistributedQueue]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$1(TestSimDistributedQueue.java:107)
   [junit4]   2>         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$489/327199925.call(Unknown Source)
   [junit4]   2>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$113/1886776189.run(Unknown Source)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Dec 15, 2017 10:56:41 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=6352, name=sdqtest--2548-thread-1, state=TIMED_WAITING, group=TGRP-TestGenericDistributedQueue]
   [junit4]   2> Dec 15, 2017 10:56:41 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=4, maxDocsPerChunk=511, blockSize=1), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=4, blockSize=1)), sim=RandomSimilarity(queryNorm=true): {}, locale=tr, timezone=US/Aleutian
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=86758376,total=481820672
   [junit4]   2> NOTE: All tests run in this JVM: [TestXIncludeConfig, TestReloadDeadlock, EnumFieldTest, AddReplicaTest, TestSQLHandler, TestSolrXml, TestLazyCores, NumericFieldsTest, TestHashQParserPlugin, TestSchemaVersionResource, MoveReplicaHDFSFailoverTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestFieldCacheSortRandom, LargeFieldTest, DocExpirationUpdateProcessorFactoryTest, TestRuleBasedAuthorizationPlugin, MoreLikeThisHandlerTest, HLLSerializationTest, PrimUtilsTest, SolrIndexConfigTest, SchemaVersionSpecificBehaviorTest, TestUninvertingReader, UpdateLogTest, TestComplexPhraseLeadingWildcard, SpellingQueryConverterTest, SolrJmxReporterTest, AnalyticsMergeStrategyTest, RollingRestartTest, MetricsHandlerTest, TestDocSet, CollectionTooManyReplicasTest, TestReplicaProperties, TestStressLucene, TestJsonRequest, BJQParserTest, TestSolrCloudWithKerberosAlt, AssignTest, TestGenericDistributedQueue]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGenericDistributedQueue -Dtests.seed=7B5E06FB9A906FD4 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=tr -Dtests.timezone=US/Aleutian -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestGenericDistributedQueue (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue: 
   [junit4]    >    1) Thread[id=6352, name=sdqtest--2548-thread-1, state=TIMED_WAITING, group=TGRP-TestGenericDistributedQueue]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.lambda$testDistributedQueueBlocking$1(TestSimDistributedQueue.java:107)
   [junit4]    >         at org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue$$Lambda$489/327199925.call(Unknown Source)
   [junit4]    >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    >         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$113/1886776189.run(Unknown Source)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7B5E06FB9A906FD4]:0)
   [junit4] Completed [106/763 (3!)] on J1 in 28.39s, 4 tests, 2 errors <<< FAILURES!

[...truncated 50356 lines...]

[JENKINS] Lucene-Solr-Tests-7.x - Build # 283 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/283/

15 tests failed.
FAILED:  org.apache.solr.cloud.AliasIntegrationTest.test

Error Message:
Collection not found: testalias6

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: testalias6
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563:42B02EFAE3EED89B]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:850)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.cloud.AliasIntegrationTest.searchSeveralWays(AliasIntegrationTest.java:296)
	at org.apache.solr.cloud.AliasIntegrationTest.searchSeveralWays(AliasIntegrationTest.java:290)
	at org.apache.solr.cloud.AliasIntegrationTest.test(AliasIntegrationTest.java:250)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CustomCollectionTest.testCreateShardRepFactor

Error Message:
Error from server at http://127.0.0.1:53816/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000004

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:53816/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000004
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563:3E0A9BA45546A0A3]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.CustomCollectionTest.testCreateShardRepFactor(CustomCollectionTest.java:187)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Could not load collection from ZK: collMinRf_1x3

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collMinRf_1x3
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563:42B02EFAE3EED89B]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
	at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
	at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1960)
	at org.apache.solr.cloud.HttpPartitionTest.testMinRf(HttpPartitionTest.java:247)
	at org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collMinRf_1x3/state.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
	... 46 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:     1) Thread[id=6922, name=jetty-launcher-1329-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)    2) Thread[id=6918, name=jetty-launcher-1329-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   1) Thread[id=6922, name=jetty-launcher-1329-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   2) Thread[id=6918, name=jetty-launcher-1329-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud

Error Message:
Error from server at http://127.0.0.1:34041/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:34041/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorRandomCloud.java:111)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testReadApi

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563:9DCDEA9596E05778]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testReadApi(AutoScalingHandlerTest.java:724)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory

Error Message:
expected:<8> but was:<10>

Stack Trace:
java.lang.AssertionError: expected:<8> but was:<10>
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563:A718B5DDF75A4A64]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory(AutoscalingHistoryHandlerTest.java:259)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestXmlQParser

Error Message:
The test or suite printed 11200 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true

Stack Trace:
java.lang.AssertionError: The test or suite printed 11200 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563]:0)
	at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:211)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.uninverting.TestFieldCache

Error Message:
The test or suite printed 15652 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true

Stack Trace:
java.lang.AssertionError: The test or suite printed 15652 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563]:0)
	at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:211)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.uninverting.TestLegacyFieldCache

Error Message:
The test or suite printed 19006 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true

Stack Trace:
java.lang.AssertionError: The test or suite printed 19006 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563]:0)
	at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:211)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.uninverting.TestUninvertingReader

Error Message:
The test or suite printed 15652 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true

Stack Trace:
java.lang.AssertionError: The test or suite printed 15652 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563]:0)
	at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:211)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:
This doc was supposed to have been deleted, but was: SolrDocument{id=1, inplace_updatable_float=1.0, _version_=1586776245469708288, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0}

Stack Trace:
java.lang.AssertionError: This doc was supposed to have been deleted, but was: SolrDocument{id=1, inplace_updatable_float=1.0, _version_=1586776245469708288, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0}
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563:42B02EFAE3EED89B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.delayedReorderingFetchesMissingUpdateFromLeaderTest(TestInPlaceUpdatesDistrib.java:933)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:145)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelUpdateStream

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
	at __randomizedtesting.SeedInfo.seed([904DB7E1B9DC20DA:F3A3F403E5E8DBB1]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
	at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1205)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:848)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
	at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.cleanIndex(StreamExpressionTest.java:114)
	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
	... 44 more


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit

Error Message:
Shard split did not succeed after a previous failed split attempt left sub-shards in construction state

Stack Trace:
java.lang.AssertionError: Shard split did not succeed after a previous failed split attempt left sub-shards in construction state
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563:33A9828F7167F8E9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit(ShardSplitTest.java:294)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testCooldown

Error Message:
[TestEvent{timestamp=1513267886192000000, config={   "trigger":"node_added_cooldown_trigger",   "afterAction":[],   "stage":[     "FAILED",     "SUCCEEDED",     "IGNORED"],   "class":"org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerListener",   "beforeAction":[]}, stage=SUCCEEDED, actionName='null', event={   "id":"1500351fb9ec0e00Tem126znorjc354qcltmurzb0y",   "source":"node_added_cooldown_trigger",   "eventTime":1513267885176000000,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[1513267885176000000],     "_enqueue_time_":1513267886188000000,     "nodeNames":["127.0.0.1:34057_solr"]}}, message='null'}]

Stack Trace:
java.lang.AssertionError: [TestEvent{timestamp=1513267886192000000, config={
  "trigger":"node_added_cooldown_trigger",
  "afterAction":[],
  "stage":[
    "FAILED",
    "SUCCEEDED",
    "IGNORED"],
  "class":"org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerListener",
  "beforeAction":[]}, stage=SUCCEEDED, actionName='null', event={
  "id":"1500351fb9ec0e00Tem126znorjc354qcltmurzb0y",
  "source":"node_added_cooldown_trigger",
  "eventTime":1513267885176000000,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[1513267885176000000],
    "_enqueue_time_":1513267886188000000,
    "nodeNames":["127.0.0.1:34057_solr"]}}, message='null'}]
	at __randomizedtesting.SeedInfo.seed([CAE411204D12B563:FB5A7CC433B8C091]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testCooldown(TriggerIntegrationTest.java:1221)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11785 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_CAE411204D12B563-001/init-core-data-001
   [junit4]   2> 1060032 WARN  (SUITE-AutoscalingHistoryHandlerTest-seed#[CAE411204D12B563]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 1060032 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[CAE411204D12B563]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1060034 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[CAE411204D12B563]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 1060034 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[CAE411204D12B563]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1060034 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[CAE411204D12B563]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_CAE411204D12B563-001/tempDir-001
   [junit4]   2> 1060034 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[CAE411204D12B563]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1060072 INFO  (Thread-951) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1060072 INFO  (Thread-951) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1060172 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[CAE411204D12B563]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:43282
   [junit4]   2> 1060173 ERROR (Thread-951) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1060313 INFO  (zkConnectionManagerCallback-458-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1060338 INFO  (jetty-launcher-455-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1062144 INFO  (jetty-launcher-455-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1063694 INFO  (jetty-launcher-455-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@20409aaa{/solr,null,AVAILABLE}
   [junit4]   2> 1063695 INFO  (jetty-launcher-455-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@553c610d{HTTP/1.1,[http/1.1]}{127.0.0.1:34759}
   [junit4]   2> 1063695 INFO  (jetty-launcher-455-thread-3) [    ] o.e.j.s.Server Started @1078376ms
   [junit4]   2> 1063695 INFO  (jetty-launcher-455-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34759}
   [junit4]   2> 1063695 ERROR (jetty-launcher-455-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1063695 INFO  (jetty-launcher-455-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 1063695 INFO  (jetty-launcher-455-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1063695 INFO  (jetty-launcher-455-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1063695 INFO  (jetty-launcher-455-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-14T15:13:30.302Z
   [junit4]   2> 1063697 INFO  (jetty-launcher-455-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@50b02cb{/solr,null,AVAILABLE}
   [junit4]   2> 1063697 INFO  (jetty-launcher-455-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@59fea1b9{HTTP/1.1,[http/1.1]}{127.0.0.1:52690}
   [junit4]   2> 1063697 INFO  (jetty-launcher-455-thread-2) [    ] o.e.j.s.Server Started @1078378ms
   [junit4]   2> 1063697 INFO  (jetty-launcher-455-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52690}
   [junit4]   2> 1063697 ERROR (jetty-launcher-455-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1063697 INFO  (jetty-launcher-455-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 1063697 INFO  (jetty-launcher-455-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1063697 INFO  (jetty-launcher-455-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1063697 INFO  (jetty-launcher-455-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-14T15:13:30.304Z
   [junit4]   2> 1063830 INFO  (jetty-launcher-455-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1063841 INFO  (jetty-launcher-455-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1d6c5d2f{/solr,null,AVAILABLE}
   [junit4]   2> 1063841 INFO  (jetty-launcher-455-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@758e52c9{HTTP/1.1,[http/1.1]}{127.0.0.1:34469}
   [junit4]   2> 1063841 INFO  (jetty-launcher-455-thread-1) [    ] o.e.j.s.Server Started @1078523ms
   [junit4]   2> 1063841 INFO  (jetty-launcher-455-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34469}
   [junit4]   2> 1063842 ERROR (jetty-launcher-455-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1063842 INFO  (jetty-launcher-455-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 1063842 INFO  (jetty-launcher-455-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1063842 INFO  (jetty-launcher-455-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1063842 INFO  (jetty-launcher-455-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-14T15:13:30.449Z
   [junit4]   2> 1063894 INFO  (zkConnectionManagerCallback-462-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1063894 INFO  (zkConnectionManagerCallback-460-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1064045 INFO  (jetty-launcher-455-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1064045 INFO  (jetty-launcher-455-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1064161 INFO  (jetty-launcher-455-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43282/solr
   [junit4]   2> 1064169 INFO  (zkConnectionManagerCallback-464-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1064209 INFO  (jetty-launcher-455-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43282/solr
   [junit4]   2> 1064233 INFO  (zkConnectionManagerCallback-468-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1064298 INFO  (jetty-launcher-455-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1064321 INFO  (jetty-launcher-455-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43282/solr
   [junit4]   2> 1064406 INFO  (zkConnectionManagerCallback-472-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1064514 INFO  (zkConnectionManagerCallback-476-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1064514 INFO  (zkConnectionManagerCallback-478-thread-1-processing-n:127.0.0.1:52690_solr) [n:127.0.0.1:52690_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1064778 INFO  (zkConnectionManagerCallback-480-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1064814 INFO  (jetty-launcher-455-thread-3) [n:127.0.0.1:34759_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1064830 INFO  (jetty-launcher-455-thread-3) [n:127.0.0.1:34759_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34759_solr
   [junit4]   2> 1064831 INFO  (jetty-launcher-455-thread-3) [n:127.0.0.1:34759_solr    ] o.a.s.c.Overseer Overseer (id=99173296158801928-127.0.0.1:34759_solr-n_0000000000) starting
   [junit4]   2> 1064849 INFO  (zkConnectionManagerCallback-483-thread-1-processing-n:127.0.0.1:34469_solr) [n:127.0.0.1:34469_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1064914 DEBUG (OverseerStateUpdate-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1064915 INFO  (jetty-launcher-455-thread-1) [n:127.0.0.1:34469_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1064916 INFO  (jetty-launcher-455-thread-1) [n:127.0.0.1:34469_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34469_solr
   [junit4]   2> 1066138 INFO  (jetty-launcher-455-thread-3) [n:127.0.0.1:34759_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34759_solr
   [junit4]   2> 1066138 DEBUG (OverseerAutoScalingTriggerThread-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 1066139 INFO  (OverseerStateUpdate-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1066141 DEBUG (OverseerAutoScalingTriggerThread-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 1066142 DEBUG (OverseerAutoScalingTriggerThread-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:34469_solr]
   [junit4]   2> 1066142 DEBUG (OverseerAutoScalingTriggerThread-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 1066142 DEBUG (OverseerAutoScalingTriggerThread-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 1066222 INFO  (jetty-launcher-455-thread-2) [n:127.0.0.1:52690_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1066224 INFO  (jetty-launcher-455-thread-2) [n:127.0.0.1:52690_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1066225 INFO  (jetty-launcher-455-thread-2) [n:127.0.0.1:52690_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52690_solr
   [junit4]   2> 1066261 INFO  (zkCallback-479-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1066319 DEBUG (OverseerAutoScalingTriggerThread-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 1066319 DEBUG (OverseerAutoScalingTriggerThread-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 1066344 DEBUG (OverseerStateUpdate-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:52690_solr"} current state version: 0
   [junit4]   2> 1066344 DEBUG (OverseerStateUpdate-99173296158801928-127.0.0.1:34759_solr-n_0000000000) [n:127.0.0.1:34759_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:52690_solr
   [junit4]   2> 1066582 INFO  (zkCallback-477-thread-1-processing-n:127.0.0.1:52690_solr) [n:127.0.0.1:52690_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1066651 INFO  (zkCallback-482-thread-1-processing-n:127.0.0.1:34469_solr) [n:127.0.0.1:34469_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1066320 DEBUG (ScheduledTrigger-747-thread-1) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:34469_solr, 127.0.0.1:34759_solr, 127.0.0.1:52690_solr]
   [junit4]   2> 1067060 INFO  (jetty-launcher-455-thread-2) [n:127.0.0.1:52690_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52690.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70bacf61
   [junit4]   2> 1067066 INFO  (jetty-launcher-455-thread-3) [n:127.0.0.1:34759_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34759.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70bacf61
   [junit4]   2> 1067126 INFO  (jetty-launcher-455-thread-2) [n:127.0.0.1:52690_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52690.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70bacf61
   [junit4]   2> 1067126 INFO  (jetty-launcher-455-thread-2) [n:127.0.0.1:52690_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52690.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70bacf61
   [junit4]   2> 1067129 INFO  (jetty-launcher-455-thread-3) [n:127.0.0.1:34759_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34759.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70bacf61
   [junit4]   2> 1067129 INFO  (jetty-launcher-455-thread-3) [n:127.0.0.1:34759_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34759.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70bacf61
   [junit4]   2> 1067130 INFO  (jetty-launcher-455-thread-3) [n:127.0.0.1:34759_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_CAE411204D12B563-001/tempDir-001/node3/.
   [junit4]   2> 1067159 INFO  (jetty-launcher-455-thread-2) [n:127.0.0.1:52690_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_CAE411204D12B563-001/tempDir-001/node2/.
   [junit4]   2> 1067610 INFO  (jetty-launcher-455-thread-1) [n:127.0.0.1:34469_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34469.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70bacf61
   [junit4]   2> 1067621 INFO  (jetty-launcher-455-thread-1) [n:127.0.0.1:34469_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34469.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70bacf61
   [junit4]   2> 1067621 INFO  (jetty-launcher-455-thread-1) [n:127.0.0.1:34469_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34469.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@70bacf61
   [junit4]   2> 1067622 INFO  (jetty-launcher-455-thread-1) [n:127.0.0.1:34469_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_CAE411204D12B563-001/tempDir-001/node1/.
   [junit4]   2> 1067749 DEBUG (ScheduledTrigger-747-thread-1) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:34469_solr, 127.0.0.1:34759_solr, 127.0.0.1:52690_solr]
   [junit4]   2> 1068234 INFO  (zkConnectionManagerCallback-488-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1068352 INFO  (zkConnectionManagerCallback-492-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1068461 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[CAE411204D12B563]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1068533 INFO  (SUITE-AutoscalingHistoryHandlerTest-seed#[CAE411204D12B563]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43282/solr ready
   [junit4]   2> 1068806 DEBUG (ScheduledTrigger-747-thread-2) [n:127.0.0.1:34759_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:34469_solr, 127.0.0.1:34759_solr, 127.0.0.1:52690_solr]
   [junit4]   2> 1069068 INFO  (qtp429445519-2512) [n:127.0.0.1:34469_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=3&name=AutoscalingHistoryHandlerTest_collection&nrtReplicas=3&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1069106 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.CreateCollectionCmd Create collection AutoscalingHistoryHandlerTest_collection
   [junit4]   2> 1069106 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.CreateCollectionCmd Only _default config set found, using it.
   [junit4]   2> 1069107 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/managed-schema to /configs/AutoscalingHistoryHandlerTest_collection/managed-schema
   [junit4]   2> 1069109 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/protwords.txt to /configs/AutoscalingHistoryHandlerTest_collection/protwords.txt
   [junit4]   2> 1069110 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/solrconfig.xml to /configs/AutoscalingHistoryHandlerTest_collection/solrconfig.xml
   [junit4]   2> 1069126 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/synonyms.txt to /configs/AutoscalingHistoryHandlerTest_collection/synonyms.txt
   [junit4]   2> 1069127 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/stopwords.txt to /configs/AutoscalingHistoryHandlerTest_collection/stopwords.txt
   [junit4]   2> 1069128 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_it.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/contractions_it.txt
   [junit4]   2> 1069145 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ca.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/contractions_ca.txt
   [junit4]   2> 1069148 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stemdict_nl.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stemdict_nl.txt
   [junit4]   2> 1069149 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_no.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_no.txt
   [junit4]   2> 1069166 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hy.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_hy.txt
   [junit4]   2> 1069167 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_id.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_id.txt
   [junit4]   2> 1069168 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hi.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_hi.txt
   [junit4]   2> 1069169 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ga.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_ga.txt
   [junit4]   2> 1069170 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_da.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_da.txt
   [junit4]   2> 1069190 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ja.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_ja.txt
   [junit4]   2> 1069191 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_pt.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_pt.txt
   [junit4]   2> 1069192 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hu.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_hu.txt
   [junit4]   2> 1069210 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_el.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_el.txt
   [junit4]   2> 1069211 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ru.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_ru.txt
   [junit4]   2> 1069211 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_tr.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_tr.txt
   [junit4]   2> 1069212 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ar.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_ar.txt
   [junit4]   2> 1069229 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/userdict_ja.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/userdict_ja.txt
   [junit4]   2> 1069230 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_eu.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_eu.txt
   [junit4]   2> 1069250 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_it.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_it.txt
   [junit4]   2> 1069251 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_cz.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_cz.txt
   [junit4]   2> 1069252 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stoptags_ja.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stoptags_ja.txt
   [junit4]   2> 1069253 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_fr.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/contractions_fr.txt
   [junit4]   2> 1069270 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_de.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_de.txt
   [junit4]   2> 1069270 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fa.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_fa.txt
   [junit4]   2> 1069271 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ga.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/contractions_ga.txt
   [junit4]   2> 1069272 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ca.txt to /configs/AutoscalingHistoryHandlerTest_collection/lang/stopwords_ca.txt
   [junit4]   2> 1069273 INFO  (OverseerThreadFactory-749-thread-1-processing-n:127.0.0.1:34759_solr) [n:127.0.0.1:34759_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_nl.txt t

[...truncated too long message...]

s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 848062 INFO  (zkCallback-83-thread-8-processing-n:127.0.0.1:60988_solr) [n:127.0.0.1:60988_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (0)
   [junit4]   2> 848063 INFO  (zkCallback-83-thread-9-processing-n:127.0.0.1:60988_solr) [n:127.0.0.1:60988_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 848063 INFO  (zkCallback-87-thread-5-processing-n:127.0.0.1:56727_solr) [n:127.0.0.1:56727_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (0)
   [junit4]   2> 848119 INFO  (jetty-closer-54-thread-3) [    ] o.a.s.c.Overseer Overseer (id=99173984903757832-127.0.0.1:58586_solr-n_0000000000) closing
   [junit4]   2> 848120 INFO  (OverseerStateUpdate-99173984903757832-127.0.0.1:58586_solr-n_0000000000) [n:127.0.0.1:58586_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58586_solr
   [junit4]   2> 848215 INFO  (coreCloseExecutor-238-thread-1) [n:127.0.0.1:60988_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n3] o.a.s.c.SolrCore [collection1_shard2_replica_n3]  CLOSING SolrCore org.apache.solr.core.SolrCore@34ac4fa0
   [junit4]   2> 848215 INFO  (coreCloseExecutor-238-thread-1) [n:127.0.0.1:60988_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard2.replica_n3, tag=883707808
   [junit4]   2> 848215 INFO  (coreCloseExecutor-238-thread-1) [n:127.0.0.1:60988_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n3] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@567122ae: rootName = solr_60988, domain = solr.core.collection1.shard2.replica_n3, service url = null, agent id = null] for registry solr.core.collection1.shard2.replica_n3 / com.codahale.metrics.MetricRegistry@76ec3dd
   [junit4]   2> 848239 INFO  (zkCallback-1756-thread-1-processing-n:127.0.0.1:60988_solr x:collection1_shard2_replica_n3 s:shard2 c:collection1 r:core_node4) [n:127.0.0.1:60988_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n3] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (0)
   [junit4]   2> 848240 INFO  (coreCloseExecutor-239-thread-1) [n:127.0.0.1:56727_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@e81536c
   [junit4]   2> 848240 INFO  (coreCloseExecutor-239-thread-1) [n:127.0.0.1:56727_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard1.replica_n1, tag=243356524
   [junit4]   2> 848240 INFO  (coreCloseExecutor-239-thread-1) [n:127.0.0.1:56727_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@629b3313: rootName = solr_56727, domain = solr.core.collection1.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.collection1.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@29318fac
   [junit4]   2> 848242 WARN  (OverseerAutoScalingTriggerThread-99173984903757832-127.0.0.1:58586_solr-n_0000000000) [n:127.0.0.1:58586_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 848263 INFO  (zkCallback-83-thread-9-processing-n:127.0.0.1:60988_solr) [n:127.0.0.1:60988_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60988_solr
   [junit4]   2> 848274 INFO  (jetty-closer-54-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@718f804f{/solr,null,UNAVAILABLE}
   [junit4]   2> 848305 INFO  (coreCloseExecutor-239-thread-1) [n:127.0.0.1:56727_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=243356524
   [junit4]   2> 848306 INFO  (jetty-closer-54-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3c846964{/solr,null,UNAVAILABLE}
   [junit4]   2> 848315 INFO  (coreCloseExecutor-238-thread-1) [n:127.0.0.1:60988_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=883707808
   [junit4]   2> 848455 INFO  (jetty-closer-54-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@f348c74{/solr,null,UNAVAILABLE}
   [junit4]   2> 848456 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1605636bd26000a, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 848483 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1605636bd26000c, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 848458 INFO  (zkCallback-87-thread-5-processing-n:127.0.0.1:56727_solr) [n:127.0.0.1:56727_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56727_solr
   [junit4]   2> 848490 INFO  (jetty-closer-54-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2185b25f{/solr,null,UNAVAILABLE}
   [junit4]   2> 848506 ERROR (SUITE-StreamExpressionTest-seed#[904DB7E1B9DC20DA]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 848506 INFO  (SUITE-StreamExpressionTest-seed#[904DB7E1B9DC20DA]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55348 55348
   [junit4]   2> 848633 INFO  (Thread-30) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55348 55348
   [junit4]   2> 848639 WARN  (Thread-30) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	100	/solr/aliases.json
   [junit4]   2> 	100	/solr/clusterprops.json
   [junit4]   2> 	14	/solr/configs/conf
   [junit4]   2> 	8	/solr/configs/ml
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	100	/solr/clusterstate.json
   [junit4]   2> 	48	/solr/collections/destinationCollection/state.json
   [junit4]   2> 	39	/solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 	16	/solr/collections/mainCorpus1/state.json
   [junit4]   2> 	15	/solr/collections/collection1/state.json
   [junit4]   2> 	15	/solr/collections/workQueue1/state.json
   [junit4]   2> 	15	/solr/collections/checkpointCollection/state.json
   [junit4]   2> 	14	/solr/collections/uknownCollection/state.json
   [junit4]   2> 	14	/solr/collections/mainCorpus/state.json
   [junit4]   2> 	12	/solr/collections/destination/state.json
   [junit4]   2> 	12	/solr/collections/destination1/state.json
   [junit4]   2> 	12	/solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 	12	/solr/collections/workQueue/state.json
   [junit4]   2> 	12	/solr/collections/modelCollection/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	100	/solr/live_nodes
   [junit4]   2> 	99	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_904DB7E1B9DC20DA-001
   [junit4]   2> Dec 14, 2017 6:19:44 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=8459, maxDocsPerChunk=7, blockSize=8), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=8459, blockSize=8)), sim=RandomSimilarity(queryNorm=false): {}, locale=nn-NO, timezone=Asia/Bishkek
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=133971568,total=439877632
   [junit4]   2> NOTE: All tests run in this JVM: [ReverseEvaluatorTest, EmpiricalDistributionEvaluatorTest, SolrSchemalessExampleTest, TestNamedListCodec, NaturalLogEvaluatorTest, FieldValueEvaluatorTest, TestZkConfigManager, SolrExampleStreamingTest, LessThanEvaluatorTest, StreamExpressionTest]
   [junit4] Completed [133/146 (1!)] on J0 in 670.07s, 139 tests, 1 error <<< FAILURES!

[...truncated 46464 lines...]

[JENKINS] Lucene-Solr-Tests-7.x - Build # 282 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/282/

8 tests failed.
FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:58252/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:39345/solr/MoveReplicaHDFSTest_failed_coll_true]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:58252/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:39345/solr/MoveReplicaHDFSTest_failed_coll_true]
	at __randomizedtesting.SeedInfo.seed([8B8AF53679C43555:214726C4CE17E085]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:990)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:306)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39345/solr/MoveReplicaHDFSTest_failed_coll_true: no servers hosting shard: shard2
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	... 46 more


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit

Error Message:
Shard split did not succeed after a previous failed split attempt left sub-shards in construction state

Stack Trace:
java.lang.AssertionError: Shard split did not succeed after a previous failed split attempt left sub-shards in construction state
	at __randomizedtesting.SeedInfo.seed([8B8AF53679C43555:72C7669945B178DF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit(ShardSplitTest.java:294)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.UnloadDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:43692/dbyu/qr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:43692/dbyu/qr
	at __randomizedtesting.SeedInfo.seed([8B8AF53679C43555:3DECAECD73858AD]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:174)
	at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 52 more


FAILED:  org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
	at __randomizedtesting.SeedInfo.seed([8B8AF53679C43555:B339D1C85E37E184]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple(AutoAddReplicasPlanActionTest.java:123)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SearchRateTriggerTest.testTrigger

Error Message:
expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([8B8AF53679C43555:E841C3B4E00B4678]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.autoscaling.SearchRateTriggerTest.testTrigger(SearchRateTriggerTest.java:101)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test

Error Message:
wrong number of events added to .system expected:<9> but was:<10>

Stack Trace:
java.lang.AssertionError: wrong number of events added to .system expected:<9> but was:<10>
	at __randomizedtesting.SeedInfo.seed([8B8AF53679C43555:3DECAECD73858AD]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test(SystemLogListenerTest.java:166)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:51758_solr wasn't created

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:51758_solr wasn't created
	at __randomizedtesting.SeedInfo.seed([8B8AF53679C43555:93307D3A77F1F8BA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration(TriggerIntegrationTest.java:930)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testCooldown

Error Message:
[TestEvent{timestamp=1513141458624000000, config={   "trigger":"node_added_cooldown_trigger",   "afterAction":[],   "stage":[     "FAILED",     "SUCCEEDED",     "IGNORED"],   "class":"org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerListener",   "beforeAction":[]}, stage=SUCCEEDED, actionName='null', event={   "id":"14ffc22376875300T8penvx4rq9b7g6d10oamkhsoe",   "source":"node_added_cooldown_trigger",   "eventTime":1513141457388000000,   "eventType":"NODEADDED",   "properties":{     "eventTimes":[1513141457388000000],     "_enqueue_time_":1513141458623000000,     "nodeNames":["127.0.0.1:48965_solr"]}}, message='null'}]

Stack Trace:
java.lang.AssertionError: [TestEvent{timestamp=1513141458624000000, config={
  "trigger":"node_added_cooldown_trigger",
  "afterAction":[],
  "stage":[
    "FAILED",
    "SUCCEEDED",
    "IGNORED"],
  "class":"org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerListener",
  "beforeAction":[]}, stage=SUCCEEDED, actionName='null', event={
  "id":"14ffc22376875300T8penvx4rq9b7g6d10oamkhsoe",
  "source":"node_added_cooldown_trigger",
  "eventTime":1513141457388000000,
  "eventType":"NODEADDED",
  "properties":{
    "eventTimes":[1513141457388000000],
    "_enqueue_time_":1513141458623000000,
    "nodeNames":["127.0.0.1:48965_solr"]}}, message='null'}]
	at __randomizedtesting.SeedInfo.seed([8B8AF53679C43555:BA3498D2076E40A7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testCooldown(TriggerIntegrationTest.java:1221)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11671 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.SystemLogListenerTest
   [junit4]   2> 134616 INFO  (SUITE-SystemLogListenerTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/init-core-data-001
   [junit4]   2> 134674 WARN  (SUITE-SystemLogListenerTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
   [junit4]   2> 134674 INFO  (SUITE-SystemLogListenerTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 134701 INFO  (SUITE-SystemLogListenerTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 134794 INFO  (SUITE-SystemLogListenerTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001
   [junit4]   2> 134800 INFO  (SUITE-SystemLogListenerTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 134820 INFO  (Thread-12) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 134828 INFO  (Thread-12) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 135493 INFO  (SUITE-SystemLogListenerTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:36358
   [junit4]   2> 136114 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 136379 ERROR (Thread-12) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 140268 INFO  (zkConnectionManagerCallback-8-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 141168 INFO  (jetty-launcher-5-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 141181 INFO  (jetty-launcher-5-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6caf3f7b{/solr,null,AVAILABLE}
   [junit4]   2> 141252 INFO  (jetty-launcher-5-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 141171 INFO  (jetty-launcher-5-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 141292 INFO  (jetty-launcher-5-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1d66a0a6{/solr,null,AVAILABLE}
   [junit4]   2> 141317 INFO  (jetty-launcher-5-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@52c2db4f{/solr,null,AVAILABLE}
   [junit4]   2> 141373 INFO  (jetty-launcher-5-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@13faf7f4{SSL,[ssl, http/1.1]}{127.0.0.1:52810}
   [junit4]   2> 141373 INFO  (jetty-launcher-5-thread-3) [    ] o.e.j.s.Server Started @163680ms
   [junit4]   2> 141373 INFO  (jetty-launcher-5-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52810}
   [junit4]   2> 141374 ERROR (jetty-launcher-5-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 141374 INFO  (jetty-launcher-5-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 141374 INFO  (jetty-launcher-5-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 141374 INFO  (jetty-launcher-5-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 141374 INFO  (jetty-launcher-5-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-13T02:35:53.594Z
   [junit4]   2> 141396 INFO  (jetty-launcher-5-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 141374 INFO  (jetty-launcher-5-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7967638e{SSL,[ssl, http/1.1]}{127.0.0.1:35199}
   [junit4]   2> 141397 INFO  (jetty-launcher-5-thread-2) [    ] o.e.j.s.Server Started @163703ms
   [junit4]   2> 141392 INFO  (jetty-launcher-5-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@61e2a34{SSL,[ssl, http/1.1]}{127.0.0.1:38749}
   [junit4]   2> 141400 INFO  (jetty-launcher-5-thread-1) [    ] o.e.j.s.Server Started @163706ms
   [junit4]   2> 141397 INFO  (jetty-launcher-5-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35199}
   [junit4]   2> 141400 INFO  (jetty-launcher-5-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38749}
   [junit4]   2> 141476 ERROR (jetty-launcher-5-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 141477 ERROR (jetty-launcher-5-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 141478 INFO  (jetty-launcher-5-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 141478 INFO  (jetty-launcher-5-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 141478 INFO  (jetty-launcher-5-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 141478 INFO  (jetty-launcher-5-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-13T02:35:53.698Z
   [junit4]   2> 141478 INFO  (jetty-launcher-5-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 141480 INFO  (jetty-launcher-5-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 141480 INFO  (jetty-launcher-5-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 141480 INFO  (jetty-launcher-5-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-13T02:35:53.700Z
   [junit4]   2> 141493 INFO  (zkConnectionManagerCallback-10-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 141686 INFO  (jetty-launcher-5-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 141841 INFO  (jetty-launcher-5-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:36358/solr
   [junit4]   2> 141880 INFO  (zkConnectionManagerCallback-12-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 141909 INFO  (jetty-launcher-5-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 141968 INFO  (jetty-launcher-5-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:36358/solr
   [junit4]   2> 142431 INFO  (zkConnectionManagerCallback-14-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 142604 INFO  (jetty-launcher-5-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 142640 INFO  (zkConnectionManagerCallback-20-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 142678 INFO  (jetty-launcher-5-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:36358/solr
   [junit4]   2> 142713 INFO  (zkConnectionManagerCallback-22-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 142853 INFO  (zkConnectionManagerCallback-26-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 143695 INFO  (zkConnectionManagerCallback-28-thread-1-processing-n:127.0.0.1:52810_solr) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 144353 INFO  (zkConnectionManagerCallback-32-thread-1-processing-n:127.0.0.1:38749_solr) [n:127.0.0.1:38749_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 144355 INFO  (zkConnectionManagerCallback-30-thread-1-processing-n:127.0.0.1:35199_solr) [n:127.0.0.1:35199_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 146576 INFO  (jetty-launcher-5-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 146617 INFO  (jetty-launcher-5-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38749_solr
   [junit4]   2> 146668 INFO  (jetty-launcher-5-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.Overseer Overseer (id=99164654647050249-127.0.0.1:38749_solr-n_0000000000) starting
   [junit4]   2> 147112 INFO  (jetty-launcher-5-thread-2) [n:127.0.0.1:35199_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 147235 INFO  (jetty-launcher-5-thread-2) [n:127.0.0.1:35199_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35199_solr
   [junit4]   2> 148005 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:38749_solr) [n:127.0.0.1:38749_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 148957 INFO  (jetty-launcher-5-thread-3) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 149056 INFO  (jetty-launcher-5-thread-3) [n:127.0.0.1:52810_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 149077 INFO  (jetty-launcher-5-thread-3) [n:127.0.0.1:52810_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52810_solr
   [junit4]   2> 149225 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:38749_solr) [n:127.0.0.1:38749_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 150920 INFO  (zkCallback-29-thread-1-processing-n:127.0.0.1:35199_solr) [n:127.0.0.1:35199_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 150992 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:52810_solr) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 151230 INFO  (jetty-launcher-5-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38749_solr
   [junit4]   2> 151395 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:52810_solr) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 151396 INFO  (zkCallback-29-thread-1-processing-n:127.0.0.1:35199_solr) [n:127.0.0.1:35199_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 151397 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:38749_solr) [n:127.0.0.1:38749_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 151895 DEBUG (OverseerAutoScalingTriggerThread-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
   [junit4]   2> 153008 DEBUG (OverseerAutoScalingTriggerThread-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
   [junit4]   2> 153696 DEBUG (OverseerAutoScalingTriggerThread-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 153697 DEBUG (OverseerAutoScalingTriggerThread-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
   [junit4]   2> 153697 DEBUG (OverseerAutoScalingTriggerThread-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
   [junit4]   2> 153737 INFO  (jetty-launcher-5-thread-3) [n:127.0.0.1:52810_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52810.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 154128 INFO  (jetty-launcher-5-thread-3) [n:127.0.0.1:52810_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52810.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 154128 INFO  (jetty-launcher-5-thread-3) [n:127.0.0.1:52810_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52810.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 154163 INFO  (jetty-launcher-5-thread-2) [n:127.0.0.1:35199_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35199.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 154200 DEBUG (OverseerAutoScalingTriggerThread-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
   [junit4]   2> 154201 DEBUG (OverseerAutoScalingTriggerThread-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
   [junit4]   2> 154202 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 154458 INFO  (jetty-launcher-5-thread-3) [n:127.0.0.1:52810_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001/node3/.
   [junit4]   2> 154714 INFO  (jetty-launcher-5-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38749.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 154739 INFO  (jetty-launcher-5-thread-2) [n:127.0.0.1:35199_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35199.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 154739 INFO  (jetty-launcher-5-thread-2) [n:127.0.0.1:35199_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35199.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 154945 INFO  (jetty-launcher-5-thread-2) [n:127.0.0.1:35199_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001/node2/.
   [junit4]   2> 155302 INFO  (jetty-launcher-5-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38749.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 155302 INFO  (jetty-launcher-5-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38749.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 155416 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 155454 INFO  (jetty-launcher-5-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001/node1/.
   [junit4]   2> 156464 DEBUG (ScheduledTrigger-40-thread-2) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 157473 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 157540 INFO  (zkConnectionManagerCallback-38-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 157692 INFO  (zkConnectionManagerCallback-42-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 157893 INFO  (SUITE-SystemLogListenerTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 157894 INFO  (SUITE-SystemLogListenerTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:36358/solr ready
   [junit4]   2> 158091 INFO  (qtp68455735-84) [n:127.0.0.1:38749_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=3&name=.system&nrtReplicas=3&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 158256 INFO  (OverseerThreadFactory-42-thread-1-processing-n:127.0.0.1:38749_solr) [n:127.0.0.1:38749_solr    ] o.a.s.c.CreateCollectionCmd Create collection .system
   [junit4]   2> 159172 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 160204 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 161204 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 162205 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 163276 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 163773 INFO  (OverseerStateUpdate-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":".system",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":".system_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:35199/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 163781 INFO  (OverseerStateUpdate-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":".system",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":".system_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:52810/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 163917 INFO  (OverseerStateUpdate-99164654647050249-127.0.0.1:38749_solr-n_0000000000) [n:127.0.0.1:38749_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":".system",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":".system_shard1_replica_n5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:38749/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 164292 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 164662 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n2&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 164705 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 164733 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n1&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 164734 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 164962 INFO  (zkCallback-29-thread-1-processing-n:127.0.0.1:35199_solr) [n:127.0.0.1:35199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 164962 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:52810_solr) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 165021 INFO  (zkCallback-29-thread-2-processing-n:127.0.0.1:35199_solr) [n:127.0.0.1:35199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 165060 INFO  (zkCallback-27-thread-2-processing-n:127.0.0.1:52810_solr) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 165292 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 165365 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=.system&newCollection=true&name=.system_shard1_replica_n5&action=CREATE&numShards=1&collection=.system&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 165365 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 165594 INFO  (zkCallback-29-thread-1-processing-n:127.0.0.1:35199_solr) [n:127.0.0.1:35199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 165594 INFO  (zkCallback-27-thread-2-processing-n:127.0.0.1:52810_solr) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 165594 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:52810_solr) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 165594 INFO  (zkCallback-29-thread-2-processing-n:127.0.0.1:35199_solr) [n:127.0.0.1:35199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 165594 INFO  (zkCallback-31-thread-1-processing-n:127.0.0.1:38749_solr) [n:127.0.0.1:38749_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 165597 INFO  (zkCallback-31-thread-2-processing-n:127.0.0.1:38749_solr) [n:127.0.0.1:38749_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 165984 WARN  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.Config You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
   [junit4]   2> 165986 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 165989 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 166094 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 166132 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 166253 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema [.system_shard1_replica_n1] Schema name=_system collection or core
   [junit4]   2> 166256 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.IndexSchema [.system_shard1_replica_n2] Schema name=_system collection or core
   [junit4]   2> 166300 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 166435 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 166459 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 166640 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 166699 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/.system/managed-schema
   [junit4]   2> 166704 WARN  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.ManagedIndexSchemaFactory The schema has been upgraded to managed, but the non-managed schema schema.xml is still loadable.  PLEASE REMOVE THIS FILE.
   [junit4]   2> 167114 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.IndexSchema [.system_shard1_replica_n5] Schema name=_system collection or core
   [junit4]   2> 167129 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/.system/schema.xml to /configs/.system/schema.xml.bak
   [junit4]   2> 167129 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n1' using configuration from collection .system, trusted=true
   [junit4]   2> 167158 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35199.solr.core..system.shard1.replica_n1' (registry 'solr.core..system.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 167158 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 167158 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [[.system_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001/node2/.system_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001/node2/./.system_shard1_replica_n1/data/]
   [junit4]   2> 167187 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 167187 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n5' using configuration from collection .system, trusted=true
   [junit4]   2> 167188 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38749.solr.core..system.shard1.replica_n5' (registry 'solr.core..system.shard1.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 167188 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 167208 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.ManagedIndexSchema Managed schema znode at /configs/.system/managed-schema already exists - no need to create it
   [junit4]   2> 167236 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, the non-managed schema /configs/.system/schema.xml no longer exists.
   [junit4]   2> 167237 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore '.system_shard1_replica_n2' using configuration from collection .system, trusted=true
   [junit4]   2> 167237 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52810.solr.core..system.shard1.replica_n2' (registry 'solr.core..system.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@14ca8def
   [junit4]   2> 167237 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 167237 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.SolrCore [[.system_shard1_replica_n5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001/node1/.system_shard1_replica_n5], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001/node1/./.system_shard1_replica_n5/data/]
   [junit4]   2> 167238 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.c.SolrCore [[.system_shard1_replica_n2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001/node3/.system_shard1_replica_n2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_8B8AF53679C43555-001/tempDir-001/node3/./.system_shard1_replica_n2/data/]
   [junit4]   2> 167392 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 168173 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 168173 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 168176 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs 
   [junit4]   2> 168176 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 169968 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 170170 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4786bfa6[.system_shard1_replica_n1] main]
   [junit4]   2> 170191 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
   [junit4]   2> 170191 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
   [junit4]   2> 170209 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
   [junit4]   2> 170211 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 170211 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 170226 INFO  (searcherExecutor-49-thread-1-processing-n:127.0.0.1:35199_solr x:.system_shard1_replica_n1 s:shard1 c:.system r:core_node3) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SolrCore [.system_shard1_replica_n1] Registered new searcher Searcher@4786bfa6[.system_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 170227 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586634510771945472
   [junit4]   2> 170459 INFO  (zkCallback-31-thread-3-processing-n:127.0.0.1:38749_solr) [n:127.0.0.1:38749_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 170459 INFO  (zkCallback-31-thread-2-processing-n:127.0.0.1:38749_solr) [n:127.0.0.1:38749_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 170461 INFO  (zkCallback-29-thread-2-processing-n:127.0.0.1:35199_solr) [n:127.0.0.1:35199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 170461 INFO  (zkCallback-29-thread-1-processing-n:127.0.0.1:35199_solr) [n:127.0.0.1:35199_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 170463 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=3 found=1 timeoutin=9999ms
   [junit4]   2> 170464 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:52810_solr) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 170464 INFO  (zkCallback-27-thread-2-processing-n:127.0.0.1:52810_solr) [n:127.0.0.1:52810_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/.system/state.json] for collection [.system] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 170668 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 170668 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 170670 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs 
   [junit4]   2> 170670 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 170732 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 170732 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 170734 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.CommitTracker Hard AutoCommit: if 1 uncommited docs 
   [junit4]   2> 170734 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 170752 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@57b3149d[.system_shard1_replica_n2] main]
   [junit4]   2> 171004 DEBUG (ScheduledTrigger-40-thread-1) [n:127.0.0.1:38749_solr    ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:52810_solr, 127.0.0.1:35199_solr, 127.0.0.1:38749_solr]
   [junit4]   2> 170776 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
   [junit4]   2> 171006 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
   [junit4]   2> 171006 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
   [junit4]   2> 171007 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.ZkIndexSchemaReader Retrieved schema version 0 from ZooKeeper
   [junit4]   2> 171011 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.IndexSchema [.system_shard1_replica_n2] Schema name=_system collection or core
   [junit4]   2> 171020 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema _system collection or core/1.1 with uniqueid field id
   [junit4]   2> 171021 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.s.ZkIndexSchemaReader Finished refreshing schema in 13 ms
   [junit4]   2> 171021 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 171022 INFO  (qtp1485976180-62) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586634511605563392
   [junit4]   2> 171027 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.SolrIndexSearcher Opening [Searcher@1a4e62c7[.system_shard1_replica_n5] main]
   [junit4]   2> 171029 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/.system
   [junit4]   2> 171029 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/.system
   [junit4]   2> 171029 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/.system/managed-schema
   [junit4]   2> 171030 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 171031 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 171114 INFO  (searcherExecutor-51-thread-1-processing-n:127.0.0.1:52810_solr x:.system_shard1_replica_n2 s:shard1 c:.system r:core_node4) [n:127.0.0.1:52810_solr c:.system s:shard1 r:core_node4 x:.system_shard1_replica_n2] o.a.s.c.SolrCore [.system_shard1_replica_n2] Registered new searcher Searcher@57b3149d[.system_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 171114 INFO  (searcherExecutor-50-thread-1-processing-n:127.0.0.1:38749_solr x:.system_shard1_replica_n5 s:shard1 c:.system r:core_node6) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.SolrCore [.system_shard1_replica_n5] Registered new searcher Searcher@1a4e62c7[.system_shard1_replica_n5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 171115 INFO  (qtp68455735-80) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586634511703080960
   [junit4]   2> 171512 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 171513 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 171513 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:35199/solr/.system_shard1_replica_n1/
   [junit4]   2> 171535 INFO  (qtp201887450-72) [n:127.0.0.1:35199_solr c:.system s:shard1 r:core_node3 x:.system_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=.system_shard1_replica_n1 url=https://127.0.0.1:35199/solr START replicas=[https://127.0.0.1:52810/solr/.system_shard1_replica_n2/, https://127.0.0.1:38749/solr/.system_shard1_replica_n5/] nUpdates=100
   [junit4]   2> 171627 INFO  (qtp68455735-83) [n:127.0.0.1:38749_solr c:.system s:shard1 r:core_node6 x:.system_shard1_replica_n5] o.a.s.c.S.Request [.system_shard1_replica_n5]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=11
   [junit4]   2> 171875 INFO  (qtp1485976180-64) [n:127.0.0.1:52810_solr c:.s

[...truncated too long message...]

Watcher.process(ZkStateReader.java:1100)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 9183172 WARN  (zkCallback-3902-thread-3-processing-n:127.0.0.1:33653_solr) [n:127.0.0.1:33653_solr    ] o.a.s.c.ZkController Unable to register nodeLost path for 127.0.0.1:58026_solr
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/nodeLost/127.0.0.1:58026_solr
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$create$7(SolrZkClient.java:364)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:364)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.lambda$registerLiveNodesListener$2(ZkController.java:879)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.lambda$refreshLiveNodes$1(ZkStateReader.java:705)
   [junit4]   2> 	at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:704)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.access$900(ZkStateReader.java:74)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.refreshAndWatch(ZkStateReader.java:1105)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.process(ZkStateReader.java:1100)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 9183175 INFO  (jetty-closer-3676-thread-8) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3a9a806a{/solr,null,UNAVAILABLE}
   [junit4]   2> 9183188 ERROR (SUITE-TriggerIntegrationTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 9183188 INFO  (SUITE-TriggerIntegrationTest-seed#[8B8AF53679C43555]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:49804 49804
   [junit4]   2> 9183328 INFO  (Thread-5985) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:49804 49804
   [junit4]   2> 9183338 WARN  (Thread-5985) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	42	/solr/aliases.json
   [junit4]   2> 	17	/solr/security.json
   [junit4]   2> 	2	/solr/configs/conf
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	42	/solr/clusterprops.json
   [junit4]   2> 	42	/solr/clusterstate.json
   [junit4]   2> 	10	/solr/collections/collection1/state.json
   [junit4]   2> 	5	/solr/autoscaling.json
   [junit4]   2> 	3	/solr/overseer_elect/election/99165231439937560-127.0.0.1:58026_solr-n_0000000005
   [junit4]   2> 	2	/solr/overseer_elect/election/99165231439937582-127.0.0.1:48965_solr-n_0000000009
   [junit4]   2> 	2	/solr/overseer_elect/election/99165231439937586-127.0.0.1:57708_solr-n_0000000010
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	42	/solr/live_nodes
   [junit4]   2> 	42	/solr/collections
   [junit4]   2> 	5	/solr/overseer/queue-work
   [junit4]   2> 	5	/solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 	5	/solr/overseer/queue
   [junit4]   2> 	5	/solr/overseer/collection-queue-work
   [junit4]   2> 	2	/solr/autoscaling/events/node_added_trigger
   [junit4]   2> 	2	/solr/autoscaling/events/node_added_trigger1
   [junit4]   2> 	2	/solr/autoscaling/events/node_lost_trigger
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B8AF53679C43555-001
   [junit4]   2> Dec 13, 2017 5:06:29 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=22344, maxDocsPerChunk=838, blockSize=3), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=22344, blockSize=3)), sim=RandomSimilarity(queryNorm=true): {}, locale=bg, timezone=US/Pacific
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=197451936,total=526909440
   [junit4]   2> NOTE: All tests run in this JVM: [TestStreamBody, TestManagedSynonymGraphFilterFactory, TestLegacyNumericRangeQueryBuilder, TestNumericRangeQuery32, HdfsBasicDistributedZk2Test, RollingRestartTest, TestDocumentBuilder, TestCoreAdminApis, TestRangeQuery, TestNonDefinedSimilarityFactory, TestUseDocValuesAsStored2, SignatureUpdateProcessorFactoryTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestCloudJSONFacetJoinDomain, TestSolrCoreSnapshots, TestScoreJoinQPScore, TestHighFrequencyDictionaryFactory, TolerantUpdateProcessorTest, TestRemoteStreaming, DocumentAnalysisRequestHandlerTest, TestOmitPositions, TestPKIAuthenticationPlugin, TestDocTermOrds, TlogReplayBufferedWhileIndexingTest, TestDelegationWithHadoopAuth, TestFieldCacheWithThreads, TestHdfsBackupRestoreCore, CircularListTest, TestClusterStateMutator, TestAtomicUpdateErrorCases, TestCustomStream, SuggesterFSTTest, HdfsChaosMonkeySafeLeaderTest, TestManagedResourceStorage, TestHttpShardHandlerFactory, TestCollectionAPI, TestDeleteCollectionOnDownNodes, AnalysisErrorHandlingTest, JavabinLoaderTest, TestCustomSort, SOLR749Test, DirectUpdateHandlerTest, SolrCloudReportersTest, TestPrepRecovery, TestAuthenticationFramework, CdcrReplicationHandlerTest, CursorMarkTest, CloudExitableDirectoryReaderTest, TestGraphTermsQParserPlugin, TestLeaderElectionZkExpiry, ZkCLITest, SuggestComponentTest, ZkSolrClientTest, TestRandomFlRTGCloud, MBeansHandlerTest, ShardRoutingTest, DistributedVersionInfoTest, TestWriterPerf, TestLuceneMatchVersion, TermVectorComponentDistributedTest, RankQueryTest, LukeRequestHandlerTest, TestLockTree, PingRequestHandlerTest, ImplicitSnitchTest, TestCustomDocTransformer, TestRawResponseWriter, TestSchemaSimilarityResource, TestRestoreCore, DistributedFacetPivotSmallAdvancedTest, ShardSplitTest, TestSolrConfigHandler, ChangedSchemaMergeTest, ConfigureRecoveryStrategyTest, SubstringBytesRefFilterTest, ShardRoutingCustomTest, TestBlobHandler, SpellingQueryConverterTest, BadIndexSchemaTest, TestExactStatsCache, TestJsonFacets, TestStressLucene, HdfsAutoAddReplicasIntegrationTest, TestSolr4Spatial, BinaryUpdateRequestHandlerTest, OpenExchangeRatesOrgProviderTest, MoreLikeThisHandlerTest, ChaosMonkeySafeLeaderTest, TestSizeLimitedDistributedMap, TestXmlQParserPlugin, TestCursorMarkWithoutUniqueKey, TemplateUpdateProcessorTest, SolrJmxReporterCloudTest, UtilsToolTest, QueryEqualityTest, TestFastLRUCache, NodeLostTriggerTest, SchemaVersionSpecificBehaviorTest, BlobRepositoryCloudTest, TestFieldCacheSort, TestBulkSchemaConcurrent, TestCollapseQParserPlugin, LegacyCloudClusterPropTest, TestExceedMaxTermLength, TestConfigSetsAPIExclusivity, OverriddenZkACLAndCredentialsProvidersTest, MoveReplicaHDFSFailoverTest, TestSortByMinMaxFunction, PolyFieldTest, DistributedFacetExistsSmallTest, TestInPlaceUpdatesStandalone, CollectionsAPIDistributedZkTest, TestCloudNestedDocsSort, ShufflingReplicaListTransformerTest, DocValuesMultiTest, PeerSyncWithIndexFingerprintCachingTest, TestXIncludeConfig, ExitableDirectoryReaderTest, TestPayloadCheckQParserPlugin, CloneFieldUpdateProcessorFactoryTest, TestJettySolrRunner, HttpPartitionTest, ParsingFieldUpdateProcessorsTest, TestRTGBase, CursorPagingTest, TestSimpleTrackingShardHandler, AssignTest, DeleteLastCustomShardedReplicaTest, DeleteShardTest, DistribCursorPagingTest, MissingSegmentRecoveryTest, MoveReplicaHDFSTest, SSLMigrationTest, SolrCLIZkUtilsTest, TestCloudPivotFacet, TestLeaderElectionWithEmptyReplica, TestPullReplicaErrorHandling, TestRebalanceLeaders, TestSSLRandomization, TestSolrCloudWithSecureImpersonation, TestStressCloudBlindAtomicUpdates, ZkFailoverTest, ComputePlanActionTest, HttpTriggerListenerTest, TriggerIntegrationTest]
   [junit4] Completed [584/752 (7!)] on J1 in 241.43s, 13 tests, 2 failures <<< FAILURES!

[...truncated 47930 lines...]

[JENKINS] Lucene-Solr-Tests-7.x - Build # 281 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/281/

3 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Could not load collection from ZK: collection2

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection2
	at __randomizedtesting.SeedInfo.seed([1FCBAD3EC7009FA3:979F92E469FCF25B]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
	at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
	at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:907)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:658)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection2/state.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
	... 49 more


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_t2","base_url":"http://127.0.0.1:42873/av_/xl","node_name":"127.0.0.1:42873_av_%2Fxl","state":"active","type":"TLOG","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/17)={   "pullReplicas":"0",   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node4":{           "core":"c8n_1x3_lf_shard1_replica_t2",           "base_url":"http://127.0.0.1:42873/av_/xl",           "node_name":"127.0.0.1:42873_av_%2Fxl",           "state":"active",           "type":"TLOG",           "leader":"true"},         "core_node5":{           "core":"c8n_1x3_lf_shard1_replica_t1",           "base_url":"http://127.0.0.1:42213/av_/xl",           "node_name":"127.0.0.1:42213_av_%2Fxl",           "state":"down",           "type":"TLOG"},         "core_node6":{           "state":"down",           "base_url":"http://127.0.0.1:33889/av_/xl",           "core":"c8n_1x3_lf_shard1_replica_t3",           "node_name":"127.0.0.1:33889_av_%2Fxl",           "type":"TLOG"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false",   "nrtReplicas":"0",   "tlogReplicas":"3"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_t2","base_url":"http://127.0.0.1:42873/av_/xl","node_name":"127.0.0.1:42873_av_%2Fxl","state":"active","type":"TLOG","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/17)={
  "pullReplicas":"0",
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node4":{
          "core":"c8n_1x3_lf_shard1_replica_t2",
          "base_url":"http://127.0.0.1:42873/av_/xl",
          "node_name":"127.0.0.1:42873_av_%2Fxl",
          "state":"active",
          "type":"TLOG",
          "leader":"true"},
        "core_node5":{
          "core":"c8n_1x3_lf_shard1_replica_t1",
          "base_url":"http://127.0.0.1:42213/av_/xl",
          "node_name":"127.0.0.1:42213_av_%2Fxl",
          "state":"down",
          "type":"TLOG"},
        "core_node6":{
          "state":"down",
          "base_url":"http://127.0.0.1:33889/av_/xl",
          "core":"c8n_1x3_lf_shard1_replica_t3",
          "node_name":"127.0.0.1:33889_av_%2Fxl",
          "type":"TLOG"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"0",
  "tlogReplicas":"3"}
	at __randomizedtesting.SeedInfo.seed([1FCBAD3EC7009FA3:979F92E469FCF25B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.test

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([1FCBAD3EC7009FA3:979F92E469FCF25B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.assertInvariants(TimeRoutedAliasUpdateProcessorTest.java:245)
	at org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.test(TimeRoutedAliasUpdateProcessorTest.java:123)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12176 lines...]
   [junit4] Suite: org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest
   [junit4]   2> 2021446 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[1FCBAD3EC7009FA3]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/init-core-data-001
   [junit4]   2> 2021447 WARN  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[1FCBAD3EC7009FA3]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 2021447 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[1FCBAD3EC7009FA3]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2021482 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[1FCBAD3EC7009FA3]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2021482 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[1FCBAD3EC7009FA3]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001
   [junit4]   2> 2021482 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[1FCBAD3EC7009FA3]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2021493 INFO  (Thread-1098) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2021493 INFO  (Thread-1098) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2021513 ERROR (Thread-1098) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2021602 INFO  (SUITE-TimeRoutedAliasUpdateProcessorTest-seed#[1FCBAD3EC7009FA3]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:37370
   [junit4]   2> 2021604 INFO  (zkConnectionManagerCallback-6160-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2021628 INFO  (jetty-launcher-6157-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2021629 INFO  (jetty-launcher-6157-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@79a60078{/solr,null,AVAILABLE}
   [junit4]   2> 2021701 INFO  (jetty-launcher-6157-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2021743 INFO  (jetty-launcher-6157-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@ff47169{/solr,null,AVAILABLE}
   [junit4]   2> 2021745 INFO  (jetty-launcher-6157-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@65dfa40d{HTTP/1.1,[http/1.1]}{127.0.0.1:46883}
   [junit4]   2> 2021745 INFO  (jetty-launcher-6157-thread-2) [    ] o.e.j.s.Server Started @2036551ms
   [junit4]   2> 2021745 INFO  (jetty-launcher-6157-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46883}
   [junit4]   2> 2021777 ERROR (jetty-launcher-6157-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2021834 INFO  (jetty-launcher-6157-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 2021834 INFO  (jetty-launcher-6157-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2021834 INFO  (jetty-launcher-6157-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2021834 INFO  (jetty-launcher-6157-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-12T16:10:35.506Z
   [junit4]   2> 2021818 INFO  (jetty-launcher-6157-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@63b2fb4b{HTTP/1.1,[http/1.1]}{127.0.0.1:36953}
   [junit4]   2> 2021835 INFO  (jetty-launcher-6157-thread-1) [    ] o.e.j.s.Server Started @2036640ms
   [junit4]   2> 2021835 INFO  (jetty-launcher-6157-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36953}
   [junit4]   2> 2021835 ERROR (jetty-launcher-6157-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2021835 INFO  (jetty-launcher-6157-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 2021835 INFO  (jetty-launcher-6157-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2021835 INFO  (jetty-launcher-6157-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2021835 INFO  (jetty-launcher-6157-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-12T16:10:35.507Z
   [junit4]   2> 2021970 INFO  (zkConnectionManagerCallback-6162-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2021980 INFO  (zkConnectionManagerCallback-6164-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2021980 INFO  (jetty-launcher-6157-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2021983 INFO  (jetty-launcher-6157-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2022017 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1604b7df91e0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2022039 INFO  (jetty-launcher-6157-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37370/solr
   [junit4]   2> 2022075 INFO  (zkConnectionManagerCallback-6168-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2022112 INFO  (jetty-launcher-6157-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37370/solr
   [junit4]   2> 2022200 INFO  (zkConnectionManagerCallback-6172-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2022237 INFO  (zkConnectionManagerCallback-6174-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2022293 INFO  (zkConnectionManagerCallback-6176-thread-1-processing-n:127.0.0.1:36953_solr) [n:127.0.0.1:36953_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2022357 INFO  (jetty-launcher-6157-thread-2) [n:127.0.0.1:46883_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2022358 INFO  (jetty-launcher-6157-thread-2) [n:127.0.0.1:46883_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46883_solr
   [junit4]   2> 2022472 INFO  (jetty-launcher-6157-thread-2) [n:127.0.0.1:46883_solr    ] o.a.s.c.Overseer Overseer (id=99162196224638981-127.0.0.1:46883_solr-n_0000000000) starting
   [junit4]   2> 2023363 INFO  (jetty-launcher-6157-thread-2) [n:127.0.0.1:46883_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46883_solr
   [junit4]   2> 2023518 INFO  (OverseerStateUpdate-99162196224638981-127.0.0.1:46883_solr-n_0000000000) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2023966 INFO  (jetty-launcher-6157-thread-2) [n:127.0.0.1:46883_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46883.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@384e3881
   [junit4]   2> 2024039 INFO  (jetty-launcher-6157-thread-2) [n:127.0.0.1:46883_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46883.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@384e3881
   [junit4]   2> 2024039 INFO  (jetty-launcher-6157-thread-2) [n:127.0.0.1:46883_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46883.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@384e3881
   [junit4]   2> 2024040 INFO  (jetty-launcher-6157-thread-2) [n:127.0.0.1:46883_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node2/.
   [junit4]   2> 2024304 INFO  (jetty-launcher-6157-thread-1) [n:127.0.0.1:36953_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2024305 INFO  (jetty-launcher-6157-thread-1) [n:127.0.0.1:36953_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2024325 INFO  (jetty-launcher-6157-thread-1) [n:127.0.0.1:36953_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36953_solr
   [junit4]   2> 2024376 INFO  (zkCallback-6175-thread-1-processing-n:127.0.0.1:36953_solr) [n:127.0.0.1:36953_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2024376 INFO  (zkCallback-6173-thread-2-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2024577 INFO  (jetty-launcher-6157-thread-1) [n:127.0.0.1:36953_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36953.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@384e3881
   [junit4]   2> 2024613 INFO  (jetty-launcher-6157-thread-1) [n:127.0.0.1:36953_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36953.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@384e3881
   [junit4]   2> 2024613 INFO  (jetty-launcher-6157-thread-1) [n:127.0.0.1:36953_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36953.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@384e3881
   [junit4]   2> 2024614 INFO  (jetty-launcher-6157-thread-1) [n:127.0.0.1:36953_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/.
   [junit4]   2> 2024904 INFO  (zkConnectionManagerCallback-6181-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2025145 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.testParse-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testParse
   [junit4]   2> 2025147 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.testParse-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testParse
   [junit4]   2> 2025270 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2025335 INFO  (zkConnectionManagerCallback-6186-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2025384 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2025384 INFO  (TEST-TimeRoutedAliasUpdateProcessorTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37370/solr ready
   [junit4]   2> 2025401 INFO  (qtp139128648-14070) [n:127.0.0.1:46883_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&name=timeConfig&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2025518 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.CreateCollectionCmd Create collection timeConfig
   [junit4]   2> 2025534 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.CreateCollectionCmd Only _default config set found, using it.
   [junit4]   2> 2025570 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/protwords.txt to /configs/timeConfig/protwords.txt
   [junit4]   2> 2025717 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/managed-schema to /configs/timeConfig/managed-schema
   [junit4]   2> 2025767 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/solrconfig.xml to /configs/timeConfig/solrconfig.xml
   [junit4]   2> 2025768 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/synonyms.txt to /configs/timeConfig/synonyms.txt
   [junit4]   2> 2025769 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/stopwords.txt to /configs/timeConfig/stopwords.txt
   [junit4]   2> 2025803 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/params.json to /configs/timeConfig/params.json
   [junit4]   2> 2025804 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_it.txt to /configs/timeConfig/lang/contractions_it.txt
   [junit4]   2> 2025805 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_no.txt to /configs/timeConfig/lang/stopwords_no.txt
   [junit4]   2> 2025818 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hy.txt to /configs/timeConfig/lang/stopwords_hy.txt
   [junit4]   2> 2025819 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stemdict_nl.txt to /configs/timeConfig/lang/stemdict_nl.txt
   [junit4]   2> 2025820 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ca.txt to /configs/timeConfig/lang/contractions_ca.txt
   [junit4]   2> 2025821 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_id.txt to /configs/timeConfig/lang/stopwords_id.txt
   [junit4]   2> 2025822 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ga.txt to /configs/timeConfig/lang/stopwords_ga.txt
   [junit4]   2> 2025839 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_da.txt to /configs/timeConfig/lang/stopwords_da.txt
   [junit4]   2> 2025840 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hi.txt to /configs/timeConfig/lang/stopwords_hi.txt
   [junit4]   2> 2025841 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ja.txt to /configs/timeConfig/lang/stopwords_ja.txt
   [junit4]   2> 2025842 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_pt.txt to /configs/timeConfig/lang/stopwords_pt.txt
   [junit4]   2> 2025856 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_hu.txt to /configs/timeConfig/lang/stopwords_hu.txt
   [junit4]   2> 2025857 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_el.txt to /configs/timeConfig/lang/stopwords_el.txt
   [junit4]   2> 2025858 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ru.txt to /configs/timeConfig/lang/stopwords_ru.txt
   [junit4]   2> 2025859 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_tr.txt to /configs/timeConfig/lang/stopwords_tr.txt
   [junit4]   2> 2025876 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ar.txt to /configs/timeConfig/lang/stopwords_ar.txt
   [junit4]   2> 2025877 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/userdict_ja.txt to /configs/timeConfig/lang/userdict_ja.txt
   [junit4]   2> 2025878 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_it.txt to /configs/timeConfig/lang/stopwords_it.txt
   [junit4]   2> 2025879 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_eu.txt to /configs/timeConfig/lang/stopwords_eu.txt
   [junit4]   2> 2025892 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_cz.txt to /configs/timeConfig/lang/stopwords_cz.txt
   [junit4]   2> 2025893 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stoptags_ja.txt to /configs/timeConfig/lang/stoptags_ja.txt
   [junit4]   2> 2025894 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_fr.txt to /configs/timeConfig/lang/contractions_fr.txt
   [junit4]   2> 2025895 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fa.txt to /configs/timeConfig/lang/stopwords_fa.txt
   [junit4]   2> 2025912 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_de.txt to /configs/timeConfig/lang/stopwords_de.txt
   [junit4]   2> 2025913 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/contractions_ga.txt to /configs/timeConfig/lang/contractions_ga.txt
   [junit4]   2> 2025914 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ca.txt to /configs/timeConfig/lang/stopwords_ca.txt
   [junit4]   2> 2025914 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_nl.txt to /configs/timeConfig/lang/stopwords_nl.txt
   [junit4]   2> 2025915 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_bg.txt to /configs/timeConfig/lang/stopwords_bg.txt
   [junit4]   2> 2025928 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_es.txt to /configs/timeConfig/lang/stopwords_es.txt
   [junit4]   2> 2025930 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_en.txt to /configs/timeConfig/lang/stopwords_en.txt
   [junit4]   2> 2025932 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_lv.txt to /configs/timeConfig/lang/stopwords_lv.txt
   [junit4]   2> 2025950 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_th.txt to /configs/timeConfig/lang/stopwords_th.txt
   [junit4]   2> 2025951 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_ro.txt to /configs/timeConfig/lang/stopwords_ro.txt
   [junit4]   2> 2025952 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fr.txt to /configs/timeConfig/lang/stopwords_fr.txt
   [junit4]   2> 2025966 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_sv.txt to /configs/timeConfig/lang/stopwords_sv.txt
   [junit4]   2> 2025966 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_fi.txt to /configs/timeConfig/lang/stopwords_fi.txt
   [junit4]   2> 2025967 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/hyphenations_ga.txt to /configs/timeConfig/lang/hyphenations_ga.txt
   [junit4]   2> 2025968 INFO  (OverseerThreadFactory-2947-thread-1-processing-n:127.0.0.1:46883_solr) [n:127.0.0.1:46883_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/_default/lang/stopwords_gl.txt to /configs/timeConfig/lang/stopwords_gl.txt
   [junit4]   2> 2026097 INFO  (OverseerStateUpdate-99162196224638981-127.0.0.1:46883_solr-n_0000000000) [n:127.0.0.1:46883_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"timeConfig",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"timeConfig_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:36953/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 2026369 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=timeConfig&newCollection=true&name=timeConfig_shard1_replica_n1&action=CREATE&numShards=1&collection=timeConfig&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2026369 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2026571 INFO  (zkCallback-6175-thread-1-processing-n:127.0.0.1:36953_solr) [n:127.0.0.1:36953_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2026571 INFO  (zkCallback-6175-thread-2-processing-n:127.0.0.1:36953_solr) [n:127.0.0.1:36953_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2027517 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0 
   [junit4]   2> 2027517 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 2027541 WARN  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/extraction/lib
   [junit4]   2> 2027541 WARN  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 2027541 WARN  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/clustering/lib
   [junit4]   2> 2027541 WARN  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 2027542 WARN  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/langid/lib
   [junit4]   2> 2027542 WARN  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 2027542 WARN  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/velocity/lib
   [junit4]   2> 2027542 WARN  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 2027576 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2027660 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.IndexSchema [timeConfig_shard1_replica_n1] Schema name=default-config
   [junit4]   2> 2028462 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 2028538 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'timeConfig_shard1_replica_n1' using configuration from collection timeConfig, trusted=true
   [junit4]   2> 2028564 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36953.solr.core.timeConfig.shard1.replica_n1' (registry 'solr.core.timeConfig.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@384e3881
   [junit4]   2> 2028564 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2028564 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [[timeConfig_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/./timeConfig_shard1_replica_n1/data/]
   [junit4]   2> 2028620 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 2028841 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2028841 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2028843 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 2028843 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2028857 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@631e3c3f[timeConfig_shard1_replica_n1] main]
   [junit4]   2> 2028875 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/timeConfig
   [junit4]   2> 2028876 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/timeConfig
   [junit4]   2> 2028876 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/timeConfig/managed-schema
   [junit4]   2> 2028892 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 2028893 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2028893 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 2028895 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2028895 INFO  (searcherExecutor-2952-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@631e3c3f[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2028895 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586595147354734592
   [junit4]   2> 2028895 INFO  (searcherExecutor-2952-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 2028895 INFO  (searcherExecutor-2952-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 2028897 INFO  (searcherExecutor-2952-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [timeConfig_shard1_replica_n1] Registered new searcher Searcher@631e3c3f[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2028978 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2028979 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2028979 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/
   [junit4]   2> 2028979 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2028979 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ has no replicas
   [junit4]   2> 2028979 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2028994 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ shard1
   [junit4]   2> 2029006 INFO  (zkCallback-6175-thread-2-processing-n:127.0.0.1:36953_solr) [n:127.0.0.1:36953_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2029006 INFO  (zkCallback-6175-thread-1-processing-n:127.0.0.1:36953_solr) [n:127.0.0.1:36953_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2029055 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2029058 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=timeConfig&newCollection=true&name=timeConfig_shard1_replica_n1&action=CREATE&numShards=1&collection=timeConfig&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2689
   [junit4]   2> 2029074 INFO  (qtp139128648-14070) [n:127.0.0.1:46883_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2029158 INFO  (zkCallback-6175-thread-2-processing-n:127.0.0.1:36953_solr) [n:127.0.0.1:36953_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2029158 INFO  (zkCallback-6175-thread-1-processing-n:127.0.0.1:36953_solr) [n:127.0.0.1:36953_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/timeConfig/state.json] for collection [timeConfig] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2029540 INFO  (OverseerCollectionConfigSetProcessor-99162196224638981-127.0.0.1:46883_solr-n_0000000000) [n:127.0.0.1:46883_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2030075 INFO  (qtp139128648-14070) [n:127.0.0.1:46883_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&name=timeConfig&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=4673
   [junit4]   2> 2030083 INFO  (qtp1818415684-14077) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Executed config commands successfully and persisted to ZK [{"set-user-property":{"timePartitionAliasName":"myalias"}}, {"set-user-property":{"update.autoCreateFields":false}}, {"add-updateprocessor":{
   [junit4]   2>     "name":"tolerant",
   [junit4]   2>     "class":"solr.TolerantUpdateProcessorFactory"}}, {"add-updateprocessor":{
   [junit4]   2>     "name":"inc",
   [junit4]   2>     "class":"org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest$IncrementURPFactory",
   [junit4]   2>     "fieldName":"integer_i"}}]
   [junit4]   2> 2030083 INFO  (qtp1818415684-14077) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Waiting up to 30 secs for 1 replicas to set the property overlay to be of version 0 for collection timeConfig
   [junit4]   2> 2030084 INFO  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrCore config update listener called for core timeConfig_shard1_replica_n1
   [junit4]   2> 2030096 INFO  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrCore core reload timeConfig_shard1_replica_n1
   [junit4]   2> 2030097 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 2030116 INFO  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0 
   [junit4]   2> 2030116 INFO  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 2030116 WARN  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/extraction/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/extraction/lib
   [junit4]   2> 2030129 WARN  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-cell-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 2030130 WARN  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/clustering/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/clustering/lib
   [junit4]   2> 2030130 WARN  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-clustering-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 2030130 WARN  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/langid/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/langid/lib
   [junit4]   2> 2030130 WARN  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-langid-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 2030130 WARN  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/velocity/lib filtered by .*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../contrib/velocity/lib
   [junit4]   2> 2030130 WARN  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrConfig Couldn't add files from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist filtered by solr-velocity-\d.*\.jar to classpath: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1/../../../../dist
   [junit4]   2> 2030136 INFO  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2030200 INFO  (qtp1818415684-14081) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 2030200 INFO  (qtp1818415684-14081) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2030200 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 2030201 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 1 attempts
   [junit4]   2> 2030201 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 2030216 INFO  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.s.IndexSchema [timeConfig_shard1_replica_n1] Schema name=default-config
   [junit4]   2> 2030315 INFO  (qtp1818415684-14082) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 2030315 INFO  (qtp1818415684-14082) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2030315 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 2 attempts
   [junit4]   2> 2030315 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 2030315 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 2030416 INFO  (qtp1818415684-14083) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 2030416 INFO  (qtp1818415684-14083) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2030417 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 3 attempts
   [junit4]   2> 2030417 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 2030418 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 2030540 INFO  (qtp1818415684-14076) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 2030576 INFO  (qtp1818415684-14076) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=36
   [junit4]   2> 2030577 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 2030605 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 4 attempts
   [junit4]   2> 2030605 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 2030723 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 2030739 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=15
   [junit4]   2> 2030739 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 2030743 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 5 attempts
   [junit4]   2> 2030743 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 2030839 INFO  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 2030863 INFO  (qtp1818415684-14081) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 2030873 INFO  (qtp1818415684-14081) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=9
   [junit4]   2> 2030874 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 6 attempts
   [junit4]   2> 2030874 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 2030885 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 2030949 INFO  (Thread-1117) [n:127.0.0.1:36953_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'timeConfig_shard1_replica_n1' using configuration from collection timeConfig
   [junit4]   2> 2031089 INFO  (qtp1818415684-14082) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 2031090 INFO  (qtp1818415684-14082) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=32
   [junit4]   2> 2031090 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 7 attempts
   [junit4]   2> 2031090 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 1 secs, maxWait 30
   [junit4]   2> 2031090 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 2031122 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36953.solr.core.timeConfig.shard1.replica_n1' (registry 'solr.core.timeConfig.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@384e3881
   [junit4]   2> 2031122 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [[timeConfig_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/timeConfig_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.update.processor.TimeRoutedAliasUpdateProcessorTest_1FCBAD3EC7009FA3-001/tempDir-001/node1/./timeConfig_shard1_replica_n1/data/]
   [junit4]   2> 2031154 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
   [junit4]   2> 2031244 INFO  (qtp1818415684-14083) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 2031246 INFO  (qtp1818415684-14083) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 2031246 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 8 attempts
   [junit4]   2> 2031246 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 1 secs, maxWait 30
   [junit4]   2> 2031246 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 2031355 INFO  (qtp1818415684-14076) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 2031355 INFO  (qtp1818415684-14076) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.S.Request [timeConfig_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2031357 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:36953/solr/timeConfig_shard1_replica_n1/ for prop overlay   after 9 attempts
   [junit4]   2> 2031357 INFO  (solrHandlerExecutor-2957-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Time elapsed : 1 secs, maxWait 30
   [junit4]   2> 2031373 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 2031374 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 2031374 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2031405 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3ec13c4d[timeConfig_shard1_replica_n1] main]
   [junit4]   2> 2031415 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/timeConfig
   [junit4]   2> 2031415 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/timeConfig
   [junit4]   2> 2031415 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/timeConfig/managed-schema
   [junit4]   2> 2031416 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 2031416 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2031416 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 2031417 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2031417 INFO  (searcherExecutor-2958-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@3ec13c4d[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2031417 INFO  (searcherExecutor-2958-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 2031417 INFO  (searcherExecutor-2958-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 2031532 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586595150119829504
   [junit4]   2> 2031535 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 2031557 INFO  (searcherExecutor-2958-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [timeConfig_shard1_replica_n1] Registered new searcher Searcher@3ec13c4d[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2031557 INFO  (Thread-1117) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1d36f0f4[timeConfig_shard1_replica_n1] main]
   [junit4]   2> 2031558 INFO  (searcherExecutor-2958-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@1d36f0f4[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2031558 INFO  (searcherExecutor-2958-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 2031573 INFO  (searcherExecutor-2958-thread-1-processing-n:127.0.0.1:36953_solr x:timeConfig_shard1_replica_n1 s:shard1 c:timeConfig r:core_node2) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.c.SolrCore [timeConfig_shard1_replica_n1] Registered new searcher Searcher@1d36f0f4[timeConfig_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2031577 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler I already have the expected version 0 of config
   [junit4]   2> 2031577 INFO  (qtp1818415684-14079) [n:127.0.0.1:36953_solr c:timeConfig s:shard1 r:core_node2 x:timeConfig_shard1_replica_n1] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
   [junit4]   2> 2031577 INFO  (qtp1818415684-14079) [n:127.0.

[...truncated too long message...]

loverAfterPartitionTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6311ec6b{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 4385410 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@341a04dd{/av_/xl,null,UNAVAILABLE}
   [junit4]   2> 4385450 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4385467 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39695 39695
   [junit4]   2> 4385719 INFO  (Thread-7358) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39695 39695
   [junit4]   2> 4385720 WARN  (Thread-7358) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/aliases.json
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 	4	/solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/clusterprops.json
   [junit4]   2> 	6	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/collections/collection1/state.json
   [junit4]   2> 	3	/solr/collections/c8n_1x3_lf/state.json
   [junit4]   2> 	2	/solr/autoscaling.json
   [junit4]   2> 	2	/solr/overseer_elect/election/99162338007449604-127.0.0.1:39737_av_%2Fxl-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/live_nodes
   [junit4]   2> 	6	/solr/collections
   [junit4]   2> 	2	/solr/overseer/queue
   [junit4]   2> 	2	/solr/overseer/collection-queue-work
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 4385720 WARN  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.c.SocketProxy Closing 0 connections to: http://127.0.0.1:42213/av_/xl, target: http://127.0.0.1:35726/av_/xl
   [junit4]   2> 4385720 WARN  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.c.SocketProxy Closing 2 connections to: http://127.0.0.1:33889/av_/xl, target: http://127.0.0.1:40150/av_/xl
   [junit4]   2> 4385720 WARN  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.c.SocketProxy Closing 3 connections to: http://127.0.0.1:39737/av_/xl, target: http://127.0.0.1:38244/av_/xl
   [junit4]   2> 4385720 WARN  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[1FCBAD3EC7009FA3]) [    ] o.a.s.c.SocketProxy Closing 6 connections to: http://127.0.0.1:42873/av_/xl, target: http://127.0.0.1:37268/av_/xl
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderFailoverAfterPartitionTest -Dtests.method=test -Dtests.seed=1FCBAD3EC7009FA3 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=NZ-CHAT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE  198s J0 | LeaderFailoverAfterPartitionTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_t2","base_url":"http://127.0.0.1:42873/av_/xl","node_name":"127.0.0.1:42873_av_%2Fxl","state":"active","type":"TLOG","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/17)={
   [junit4]    >   "pullReplicas":"0",
   [junit4]    >   "replicationFactor":"1",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{
   [junit4]    >         "core_node4":{
   [junit4]    >           "core":"c8n_1x3_lf_shard1_replica_t2",
   [junit4]    >           "base_url":"http://127.0.0.1:42873/av_/xl",
   [junit4]    >           "node_name":"127.0.0.1:42873_av_%2Fxl",
   [junit4]    >           "state":"active",
   [junit4]    >           "type":"TLOG",
   [junit4]    >           "leader":"true"},
   [junit4]    >         "core_node5":{
   [junit4]    >           "core":"c8n_1x3_lf_shard1_replica_t1",
   [junit4]    >           "base_url":"http://127.0.0.1:42213/av_/xl",
   [junit4]    >           "node_name":"127.0.0.1:42213_av_%2Fxl",
   [junit4]    >           "state":"down",
   [junit4]    >           "type":"TLOG"},
   [junit4]    >         "core_node6":{
   [junit4]    >           "state":"down",
   [junit4]    >           "base_url":"http://127.0.0.1:33889/av_/xl",
   [junit4]    >           "core":"c8n_1x3_lf_shard1_replica_t3",
   [junit4]    >           "node_name":"127.0.0.1:33889_av_%2Fxl",
   [junit4]    >           "type":"TLOG"}}}},
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false",
   [junit4]    >   "nrtReplicas":"0",
   [junit4]    >   "tlogReplicas":"3"}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1FCBAD3EC7009FA3:979F92E469FCF25B]:0)
   [junit4]    > 	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
   [junit4]    > 	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_1FCBAD3EC7009FA3-001
   [junit4]   2> Dec 12, 2017 4:49:56 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {range_facet_l_dv=PostingsFormat(name=MockRandom), multiDefault=PostingsFormat(name=LuceneVarGapFixedInterval), a_t=PostingsFormat(name=LuceneVarGapFixedInterval), intDefault=FST50, id_i1=FST50, id=PostingsFormat(name=MockRandom), range_facet_i_dv=PostingsFormat(name=LuceneVarGapFixedInterval), text=Lucene50(blocksize=128), range_facet_l=PostingsFormat(name=LuceneVarGapFixedInterval), timestamp=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Memory), intDefault=DocValuesFormat(name=Memory), id_i1=DocValuesFormat(name=Memory), range_facet_i_dv=DocValuesFormat(name=Lucene70), intDvoDefault=DocValuesFormat(name=Asserting), range_facet_l=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1655, maxMBSortInHeap=5.816733835577459, sim=RandomSimilarity(queryNorm=true): {}, locale=mk, timezone=NZ-CHAT
   [junit4]   2> NOTE: Linux 4.4.0-83-generic amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=4,threads=1,free=252011168,total=524812288
   [junit4]   2> NOTE: All tests run in this JVM: [MergeStrategyTest, TestOverriddenPrefixQueryForCustomFieldType, TestUseDocValuesAsStored, TestRecovery, TermsComponentTest, TestScoreJoinQPScore, ResourceLoaderTest, TestBadConfig, ConjunctionSolrSpellCheckerTest, TestConfigSetsAPI, ResponseLogComponentTest, TestIndexingPerformance, SuggestComponentContextFilterQueryTest, CleanupOldIndexTest, SimpleCollectionCreateDeleteTest, TestUtils, PrimitiveFieldTypeTest, BlockJoinFacetRandomTest, TestRetrieveFieldsOptimizer, BasicDistributedZkTest, TestLeaderElectionWithEmptyReplica, FileUtilsTest, TestDefaultStatsCache, TestBlendedInfixSuggestions, CollectionsAPISolrJTest, TestFieldTypeCollectionResource, WrapperMergePolicyFactoryTest, TestRTimerTree, ZkControllerTest, TestConfig, TestFilteredDocIdSet, CachingDirectoryFactoryTest, TestStandardQParsers, TestComplexPhraseQParserPlugin, TestUtilizeNode, TestWordDelimiterFilterFactory, CollectionStateFormat2Test, TestFastOutputStream, TestDelegationWithHadoopAuth, UpdateParamsTest, BigEndianAscendingWordSerializerTest, LeaderInitiatedRecoveryOnShardRestartTest, TestInPlaceUpdatesStandalone, DistributedExpandComponentTest, TestSearcherReuse, TestSolrDeletionPolicy1, MultiTermTest, QueryResultKeyTest, DocValuesTest, TestLuceneIndexBackCompat, TestUpdate, TermVectorComponentTest, TestReplicationHandler, ParsingFieldUpdateProcessorsTest, TestUnifiedSolrHighlighter, PeerSyncTest, TestCloudPseudoReturnFields, TestNonDefinedSimilarityFactory, TestSchemaResource, SparseHLLTest, TestFieldCacheWithThreads, TestPhraseSuggestions, ClassificationUpdateProcessorFactoryTest, SolrXmlInZkTest, SpatialRPTFieldTypeTest, TestCloudNestedDocsSort, TestSolrCloudSnapshots, SearchHandlerTest, StressHdfsTest, TestDistributedSearch, PropertiesRequestHandlerTest, TestExceedMaxTermLength, TestTlogReplica, TestSQLHandlerNonCloud, DistributedFacetPivotWhiteBoxTest, BooleanFieldTest, TestFoldingMultitermQuery, TestQueryWrapperFilter, DateFieldTest, TestJoin, NodeAddedTriggerTest, TestSolrJ, TestSweetSpotSimilarityFactory, TestRequestStatusCollectionAPI, FullHLLTest, TestRebalanceLeaders, TestSystemCollAutoCreate, SolrCoreTest, CacheHeaderTest, StatsReloadRaceTest, SpellCheckComponentTest, TestConfigSetImmutable, ChaosMonkeyNothingIsSafeWithPullReplicasTest, SolrSlf4jReporterTest, TestSolrCloudWithDelegationTokens, StatelessScriptUpdateProcessorFactoryTest, TestCorePropertiesReload, ExecutePlanActionTest, LeaderFailoverAfterPartitionTest]
   [junit4] Completed [364/751 (3!)] on J0 in 198.92s, 1 test, 1 failure <<< FAILURES!

[...truncated 48703 lines...]

[JENKINS] Lucene-Solr-Tests-7.x - Build # 280 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/280/

4 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: collection1
	at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E:1B47F0B7461293A6]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
	at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
	at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:110)
	at org.apache.solr.cloud.ChaosMonkey.logCollectionStateSummary(ChaosMonkey.java:710)
	at org.apache.solr.cloud.ChaosMonkey.wait(ChaosMonkey.java:704)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:172)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:     1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5913, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)    6) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=5883, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    8) Thread[id=5882, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    9) Thread[id=5884, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:502)         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)         at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)         at org.apache.solr.common.cloud.SolrZkClient$$Lambda$222/2079815092.execute(Unknown Source)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)         at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)         at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)         at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1205)         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:848)         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest: 
   1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5913, name=StoppableCommitThread-EventThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
   6) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=5883, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   8) Thread[id=5882, name=StoppableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   9) Thread[id=5884, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
        at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.SolrZkClient$$Lambda$222/2079815092.execute(Unknown Source)
        at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
        at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
        at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
        at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1205)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:848)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
        at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
        at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
	at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)    3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=5883, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.lang.Object.hashCode(Native Method)         at java.util.WeakHashMap.hash(WeakHashMap.java:298)         at java.util.WeakHashMap.getEntry(WeakHashMap.java:426)         at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)         at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)         at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)    7) Thread[id=5882, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]         at java.security.AccessController.doPrivileged(Native Method)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)         at java.io.PrintStream.flush(PrintStream.java:338)         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at java.io.PrintStream.write(PrintStream.java:482)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.write(PrintStream.java:527)         at java.io.PrintStream.print(PrintStream.java:669)         at java.io.PrintStream.println(PrintStream.java:823)         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at java.lang.Throwable.printStackTrace(Throwable.java:658)         at java.lang.Throwable.printStackTrace(Throwable.java:643)         at java.lang.Throwable.printStackTrace(Throwable.java:634)         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=5883, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.lang.Object.hashCode(Native Method)
        at java.util.WeakHashMap.hash(WeakHashMap.java:298)
        at java.util.WeakHashMap.getEntry(WeakHashMap.java:426)
        at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
        at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
        at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
        at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   7) Thread[id=5882, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
        at java.security.AccessController.doPrivileged(Native Method)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:472)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:482)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.write(PrintStream.java:527)
        at java.io.PrintStream.print(PrintStream.java:669)
        at java.io.PrintStream.println(PrintStream.java:823)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
	at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E]:0)


FAILED:  org.apache.solr.handler.TestSolrConfigHandlerCloud.test

Error Message:
{   "responseHeader":{     "status":500,     "QTime":34105},   "errorMessages":["4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]\n"],   "WARNING":"This response format is experimental.  It is likely to change in the future.",   "error":{     "metadata":[       "error-class","org.apache.solr.common.SolrException",       "root-error-class","org.apache.solr.common.SolrException"],     "msg":"4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]",     "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",     "code":500}} 

Stack Trace:
java.lang.AssertionError: {
  "responseHeader":{
    "status":500,
    "QTime":34105},
  "errorMessages":["4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]\n"],
  "WARNING":"This response format is experimental.  It is likely to change in the future.",
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]",
    "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 3 within 30 seconds! Failed cores: [https://127.0.0.1:38626/collection1_shard2_replica_n41/, https://127.0.0.1:32825/collection1_shard2_replica_n45/, https://127.0.0.1:39531/collection1_shard1_replica_n43/, https://127.0.0.1:35991/collection1_shard1_replica_n47/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",
    "code":500}}

	at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E:1B47F0B7461293A6]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.core.TestSolrConfigHandler.runConfigCommand(TestSolrConfigHandler.java:185)
	at org.apache.solr.core.TestSolrConfigHandler.reqhandlertests(TestSolrConfigHandler.java:253)
	at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqHandlerAPIs(TestSolrConfigHandlerCloud.java:97)
	at org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11732 lines...]
   [junit4] Suite: org.apache.solr.handler.TestSolrConfigHandlerCloud
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/init-core-data-001
   [junit4]   2> 360963 WARN  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 360964 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 360991 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 360992 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 360993 INFO  (SUITE-TestSolrConfigHandlerCloud-seed#[9313CF6DE8EEFE5E]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 361036 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 361487 INFO  (Thread-92) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 361487 INFO  (Thread-92) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 361524 ERROR (Thread-92) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 361591 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ZkTestServer start zk server on port:38165
   [junit4]   2> 361879 INFO  (zkConnectionManagerCallback-113-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 361881 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x160467a17a00000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 361953 INFO  (zkConnectionManagerCallback-115-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 362016 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 362024 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 362032 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 362038 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 362051 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 362052 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 362057 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 362116 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 362117 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 362118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 362124 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 363269 INFO  (zkConnectionManagerCallback-118-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 363367 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 365062 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 365116 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@79ba0101{/,null,AVAILABLE}
   [junit4]   2> 365117 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6a865527{SSL,[ssl, http/1.1]}{127.0.0.1:53547}
   [junit4]   2> 365117 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.Server Started @382705ms
   [junit4]   2> 365117 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/tempDir-001/control/data, hostContext=/, hostPort=53547, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/cores}
   [junit4]   2> 365118 ERROR (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 365118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 365118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 365118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 365118 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-11T16:48:18.351Z
   [junit4]   2> 365125 INFO  (zkConnectionManagerCallback-120-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 365148 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 365148 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/solr.xml
   [junit4]   2> 366059 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 366059 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 366062 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 366128 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38165/solr
   [junit4]   2> 366353 INFO  (zkConnectionManagerCallback-124-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 366388 INFO  (zkConnectionManagerCallback-126-thread-1-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 367191 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 367192 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53547_
   [junit4]   2> 367207 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.Overseer Overseer (id=99156681998467077-127.0.0.1:53547_-n_0000000000) starting
   [junit4]   2> 367359 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53547_
   [junit4]   2> 367433 INFO  (zkCallback-125-thread-1-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 371663 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 371834 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 371834 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 372162 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:53547_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/cores
   [junit4]   2> 373323 INFO  (zkConnectionManagerCallback-132-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 373324 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 373399 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38165/solr ready
   [junit4]   2> 374082 INFO  (qtp867239183-513) [n:127.0.0.1:53547_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:53547_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 374107 INFO  (OverseerThreadFactory-217-thread-1-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 374584 INFO  (qtp867239183-510) [n:127.0.0.1:53547_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 374584 INFO  (qtp867239183-510) [n:127.0.0.1:53547_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 374744 INFO  (zkCallback-125-thread-1-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 375884 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 376356 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 378385 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 378659 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 378660 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 378660 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 378660 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 378862 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=920052104, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 378868 WARN  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 379343 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 379343 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 379352 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 379352 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 379407 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=8.2099609375, floorSegmentMB=0.40625, forceMergeDeletesPctAllowed=3.600584705322606, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15346715754849552
   [junit4]   2> 379415 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@39935aaa[control_collection_shard1_replica_n1] main]
   [junit4]   2> 379416 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 379417 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 379417 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 379419 INFO  (searcherExecutor-220-thread-1-processing-n:127.0.0.1:53547_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@39935aaa[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 379423 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586506930749177856
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:53547/control_collection_shard1_replica_n1/
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:53547/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 379479 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 379531 INFO  (zkCallback-125-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 379543 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:53547/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 379546 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 379575 INFO  (qtp867239183-510) [n:127.0.0.1:53547_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=4991
   [junit4]   2> 379620 INFO  (qtp867239183-513) [n:127.0.0.1:53547_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 379663 INFO  (zkCallback-125-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 380284 INFO  (OverseerCollectionConfigSetProcessor-99156681998467077-127.0.0.1:53547_-n_0000000000) [n:127.0.0.1:53547_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 380643 INFO  (qtp867239183-513) [n:127.0.0.1:53547_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:53547_&wt=javabin&version=2} status=0 QTime=6561
   [junit4]   2> 380927 INFO  (zkConnectionManagerCallback-136-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 380928 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 380929 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38165/solr ready
   [junit4]   2> 380929 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 381078 INFO  (qtp867239183-511) [n:127.0.0.1:53547_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 381103 INFO  (OverseerThreadFactory-217-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 381112 WARN  (OverseerThreadFactory-217-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 381252 INFO  (qtp867239183-511) [n:127.0.0.1:53547_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 381259 INFO  (qtp867239183-511) [n:127.0.0.1:53547_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=181
   [junit4]   2> 381267 INFO  (OverseerCollectionConfigSetProcessor-99156681998467077-127.0.0.1:53547_-n_0000000000) [n:127.0.0.1:53547_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 384021 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001 of type NRT
   [junit4]   2> 384029 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 384301 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@47dfed71{/,null,AVAILABLE}
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6336bd1c{SSL,[ssl, http/1.1]}{127.0.0.1:38626}
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.e.j.s.Server Started @401890ms
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=38626, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/cores}
   [junit4]   2> 384302 ERROR (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 384302 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-11T16:48:37.535Z
   [junit4]   2> 384479 INFO  (zkConnectionManagerCallback-138-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 384495 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 384495 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/solr.xml
   [junit4]   2> 384532 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 384532 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 384533 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 384620 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38165/solr
   [junit4]   2> 384720 INFO  (zkConnectionManagerCallback-142-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 384880 INFO  (zkConnectionManagerCallback-144-thread-1-processing-n:127.0.0.1:38626_) [n:127.0.0.1:38626_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 384915 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 384957 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 384996 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38626_
   [junit4]   2> 384997 INFO  (zkCallback-125-thread-2-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 385316 INFO  (zkCallback-135-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 385316 INFO  (zkCallback-143-thread-1-processing-n:127.0.0.1:38626_) [n:127.0.0.1:38626_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 385960 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 386077 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 386077 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 386176 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [n:127.0.0.1:38626_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/cores
   [junit4]   2> 388185 INFO  (qtp1715011692-565) [n:127.0.0.1:38626_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:38626_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 388221 INFO  (OverseerThreadFactory-217-thread-3-processing-n:127.0.0.1:53547_) [n:127.0.0.1:53547_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:38626_ for creating new replica
   [junit4]   2> 388533 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 388533 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 388846 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 389471 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
   [junit4]   2> 390571 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 390652 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
   [junit4]   2> 390652 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd872c
   [junit4]   2> 390652 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 390652 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.handler.TestSolrConfigHandlerCloud_9313CF6DE8EEFE5E-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
   [junit4]   2> 390696 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=920052104, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 390717 WARN  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 391235 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 391235 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 391236 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 391236 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 391521 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=8.2099609375, floorSegmentMB=0.40625, forceMergeDeletesPctAllowed=3.600584705322606, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15346715754849552
   [junit4]   2> 391563 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@49f56524[collection1_shard2_replica_n41] main]
   [junit4]   2> 391564 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 391565 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 391566 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 391567 INFO  (searcherExecutor-231-thread-1-processing-n:127.0.0.1:38626_ x:collection1_shard2_replica_n41 s:shard2 c:collection1) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@49f56524[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 391567 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586506943483084800
   [junit4]   2> 391604 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 391604 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 391605 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38626/collection1_shard2_replica_n41/
   [junit4]   2> 391605 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 391605 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy https://127.0.0.1:38626/collection1_shard2_replica_n41/ has no replicas
   [junit4]   2> 391605 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 391622 INFO  (zkCallback-143-thread-1-processing-n:127.0.0.1:38626_) [n:127.0.0.1:38626_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 391623 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38626/collection1_shard2_replica_n41/ shard2
   [junit4]   2> 391639 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 391641 INFO  (qtp1715011692-562) [n:127.0.0.1:38626_ c:collection1 s:shard2  x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3108
   [junit4]   2> 391653 INFO  (qtp1715011692-565) [n:127.0.0.1:38626_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:38626_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=3467
   [junit4]   2> 391759 INFO  (zkCallback-143-thread-1-processing-n:127.0.0.1:38626_) [n:127.0.0.1:38626_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 392844 INFO  (OverseerCollectionConfigSetProcessor-99156681998467077-127.0.0.1:53547_-n_0000000000) [n:127.0.0.1:53547_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 393729 INFO  (TEST-TestSolrConfigHandlerCloud.test-seed#[9313CF6DE8EEFE5E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkin

[...truncated too long message...]

casTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   [junit4]    >    9) Thread[id=5884, name=StoppableCommitThread, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:502)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
   [junit4]    >         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient$$Lambda$222/2079815092.execute(Unknown Source)
   [junit4]    >         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]    >         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
   [junit4]    >         at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
   [junit4]    >         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
   [junit4]    >         at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1205)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:848)
   [junit4]    >         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
   [junit4]    >         at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:524)
   [junit4]    >         at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:544)
   [junit4]    >         at org.apache.solr.cloud.StoppableCommitThread.run(StoppableCommitThread.java:49)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=5915, name=zkCallback-2015-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    2) Thread[id=5912, name=StoppableCommitThread-SendThread(127.0.0.1:52267), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
   [junit4]    >    3) Thread[id=5951, name=zkCallback-2015-thread-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=5914, name=zkConnectionManagerCallback-2016-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    5) Thread[id=5950, name=zkCallback-2015-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    6) Thread[id=5883, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.lang.Object.hashCode(Native Method)
   [junit4]    >         at java.util.WeakHashMap.hash(WeakHashMap.java:298)
   [junit4]    >         at java.util.WeakHashMap.getEntry(WeakHashMap.java:426)
   [junit4]    >         at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
   [junit4]    >         at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
   [junit4]    >         at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
   [junit4]    >         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:746)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:135)
   [junit4]    >    7) Thread[id=5882, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
   [junit4]    >         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
   [junit4]    >         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4]    >         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.flush(PrintStream.java:338)
   [junit4]    >         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.write(PrintStream.java:482)
   [junit4]    >         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4]    >         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
   [junit4]    >         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
   [junit4]    >         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
   [junit4]    >         at java.io.PrintStream.write(PrintStream.java:527)
   [junit4]    >         at java.io.PrintStream.print(PrintStream.java:669)
   [junit4]    >         at java.io.PrintStream.println(PrintStream.java:823)
   [junit4]    >         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
   [junit4]    >         at java.lang.Throwable.printStackTrace(Throwable.java:658)
   [junit4]    >         at java.lang.Throwable.printStackTrace(Throwable.java:643)
   [junit4]    >         at java.lang.Throwable.printStackTrace(Throwable.java:634)
   [junit4]    >         at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:121)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9313CF6DE8EEFE5E]:0)
   [junit4] Completed [137/751 (2!)] on J1 in 99.00s, 1 test, 3 errors <<< FAILURES!

[...truncated 49397 lines...]

[JENKINS] Lucene-Solr-Tests-7.x - Build # 279 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/279/

7 tests failed.
FAILED:  org.apache.lucene.classification.BooleanPerceptronClassifierTest.testPerformance

Error Message:
training took more than 10s: 10s

Stack Trace:
java.lang.AssertionError: training took more than 10s: 10s
	at __randomizedtesting.SeedInfo.seed([EA7BEF4AA32652D4:2D9A1D68C8926A7B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.lucene.classification.BooleanPerceptronClassifierTest.testPerformance(BooleanPerceptronClassifierTest.java:93)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:A493FE78D98F17A0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit(ShardSplitTest.java:279)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCloudSearcherWarming.testRepFactor1LeaderStartup

Error Message:
Error from server at http://127.0.0.1:36410/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000012

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:36410/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000012
	at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:8AF6200FF41A13C4]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.TestCloudSearcherWarming.testRepFactor1LeaderStartup(TestCloudSearcherWarming.java:100)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCollectionsAPIViaSolrCloudCluster.testStopAllStartAll

Error Message:
Could not load collection from ZK: testStopAllStartAllCollection

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: testStopAllStartAllCollection
	at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:2BE072A4A4CDF705]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
	at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
	at org.apache.solr.common.cloud.ZkStateReader.forceUpdateCollection(ZkStateReader.java:365)
	at org.apache.solr.cloud.TestCollectionsAPIViaSolrCloudCluster.testStopAllStartAll(TestCollectionsAPIViaSolrCloudCluster.java:231)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testStopAllStartAllCollection/state.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
	... 42 more


FAILED:  org.apache.solr.cloud.TestCryptoKeys.test

Error Message:
{   "responseHeader":{     "status":500,     "QTime":30314},   "errorMessages":["4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]\n"],   "WARNING":"This response format is experimental.  It is likely to change in the future.",   "error":{     "metadata":[       "error-class","org.apache.solr.common.SolrException",       "root-error-class","org.apache.solr.common.SolrException"],     "msg":"4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]",     "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",     "code":500}} 

Stack Trace:
java.lang.AssertionError: {
  "responseHeader":{
    "status":500,
    "QTime":30314},
  "errorMessages":["4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]\n"],
  "WARNING":"This response format is experimental.  It is likely to change in the future.",
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]",
    "trace":"org.apache.solr.common.SolrException: 4 out of 5 the property overlay to be of version 0 within 30 seconds! Failed cores: [https://127.0.0.1:39353/collection1_shard1_replica_n21/, https://127.0.0.1:60432/collection1_shard1_replica_n23/, https://127.0.0.1:55965/collection1_shard1_replica_n25/, https://127.0.0.1:56083/collection1_shard1_replica_n27/]\n\tat org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:771)\n\tat org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:502)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:748)\n",
    "code":500}}

	at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:D58A520D4B0637D2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.core.TestSolrConfigHandler.runConfigCommand(TestSolrConfigHandler.java:185)
	at org.apache.solr.cloud.TestCryptoKeys.test(TestCryptoKeys.java:127)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorCloud

Error Message:
Error from server at https://127.0.0.1:34960/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:34960/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000
	at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorCloud.java:121)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test

Error Message:
wrong number of events added to .system expected:<9> but was:<10>

Stack Trace:
java.lang.AssertionError: wrong number of events added to .system expected:<9> but was:<10>
	at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A:D58A520D4B0637D2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.autoscaling.SystemLogListenerTest.test(SystemLogListenerTest.java:166)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 6151 lines...]
   [junit4] Suite: org.apache.lucene.classification.BooleanPerceptronClassifierTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BooleanPerceptronClassifierTest -Dtests.method=testPerformance -Dtests.seed=EA7BEF4AA32652D4 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=es-CO -Dtests.timezone=Pacific/Yap -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 15.5s J1 | BooleanPerceptronClassifierTest.testPerformance <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: training took more than 10s: 10s
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EA7BEF4AA32652D4:2D9A1D68C8926A7B]:0)
   [junit4]    > 	at org.apache.lucene.classification.BooleanPerceptronClassifierTest.testPerformance(BooleanPerceptronClassifierTest.java:93)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=true): {bool=IB SPL-D1, cat=ClassicSimilarity, text=LM Jelinek-Mercer(0.700000)}, locale=es-CO, timezone=Pacific/Yap
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=206667792,total=253231104
   [junit4]   2> NOTE: All tests run in this JVM: [DocToDoubleVectorUtilsTest, BooleanPerceptronClassifierTest]
   [junit4] Completed [3/11 (1!)] on J1 in 18.08s, 4 tests, 1 failure <<< FAILURES!

[...truncated 5920 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestCryptoKeys
   [junit4]   2> 2222018 INFO  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/init-core-data-001
   [junit4]   2> 2222019 WARN  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=11 numCloses=11
   [junit4]   2> 2222019 INFO  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2222078 INFO  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2222080 INFO  (SUITE-TestCryptoKeys-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2222238 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2222633 INFO  (Thread-955) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2222633 INFO  (Thread-955) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2222646 ERROR (Thread-955) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2222733 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkTestServer start zk server on port:43531
   [junit4]   2> 2222734 INFO  (zkConnectionManagerCallback-653-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2223301 INFO  (zkConnectionManagerCallback-655-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2223313 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2223321 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2223329 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2223333 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2223337 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2223341 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2223369 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2223373 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2223385 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2223393 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2223401 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2223522 INFO  (zkConnectionManagerCallback-658-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2223533 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1603f0fb0d10002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2223541 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 2225244 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2225253 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7e923269{/,null,AVAILABLE}
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3041241b{SSL,[ssl, http/1.1]}{127.0.0.1:37799}
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server Started @2243105ms
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/tempDir-001/control/data, hostContext=/, hostPort=37799, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/cores}
   [junit4]   2> 2225255 ERROR (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 2225255 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2225256 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2225256 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-10T06:14:43.727Z
   [junit4]   2> 2225457 INFO  (zkConnectionManagerCallback-660-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2225488 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2225488 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/solr.xml
   [junit4]   2> 2225492 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2225492 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2225493 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2225553 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43531/solr
   [junit4]   2> 2225661 INFO  (zkConnectionManagerCallback-664-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2225841 INFO  (zkConnectionManagerCallback-666-thread-1-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2226281 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2226289 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37799_
   [junit4]   2> 2226290 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.Overseer Overseer (id=99148528425697285-127.0.0.1:37799_-n_0000000000) starting
   [junit4]   2> 2227349 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37799_
   [junit4]   2> 2227440 INFO  (zkCallback-665-thread-1-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2228432 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2228544 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2228545 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2228696 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:37799_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/cores
   [junit4]   2> 2228822 INFO  (zkConnectionManagerCallback-672-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2228823 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2228824 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43531/solr ready
   [junit4]   2> 2230711 INFO  (qtp1677638809-3865) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37799_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2230785 INFO  (OverseerThreadFactory-2045-thread-1-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 2231206 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2231207 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2231361 INFO  (zkCallback-665-thread-1-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2234704 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2234761 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1829627340
   [junit4]   2> 2234761 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1982841081
   [junit4]   2> 2234761 ERROR (Finalizer) [    ] o.a.s.c.CoreContainer CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=1827606894
   [junit4]   2> 2234981 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 2237604 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2237755 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 2237756 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2237756 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2237756 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 2237760 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=88.177734375, floorSegmentMB=1.3203125, forceMergeDeletesPctAllowed=26.2332621940895, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2239419 WARN  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2239844 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2239861 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2239862 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2239862 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2239894 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2239905 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@aec20bc[control_collection_shard1_replica_n1] main]
   [junit4]   2> 2239907 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2239907 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2239908 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2239909 INFO  (searcherExecutor-2048-thread-1-processing-n:127.0.0.1:37799_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@aec20bc[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2239925 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586376472942084096
   [junit4]   2> 2239962 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2239962 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2239962 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:37799/control_collection_shard1_replica_n1/
   [junit4]   2> 2239963 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2239963 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:37799/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 2239963 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2240002 INFO  (zkCallback-665-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2240002 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:37799/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 2240003 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2240026 INFO  (qtp1677638809-3860) [n:127.0.0.1:37799_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=8819
   [junit4]   2> 2240090 INFO  (qtp1677638809-3865) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2240277 INFO  (zkCallback-665-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2240939 INFO  (OverseerCollectionConfigSetProcessor-99148528425697285-127.0.0.1:37799_-n_0000000000) [n:127.0.0.1:37799_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2241105 INFO  (qtp1677638809-3865) [n:127.0.0.1:37799_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37799_&wt=javabin&version=2} status=0 QTime=10394
   [junit4]   2> 2241866 INFO  (zkConnectionManagerCallback-676-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2241867 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2241868 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43531/solr ready
   [junit4]   2> 2241868 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2241906 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2241939 INFO  (OverseerThreadFactory-2045-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 2241940 WARN  (OverseerThreadFactory-2045-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 2242289 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2242289 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=383
   [junit4]   2> 2243982 INFO  (OverseerCollectionConfigSetProcessor-99148528425697285-127.0.0.1:37799_-n_0000000000) [n:127.0.0.1:37799_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2244242 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001 of type NRT
   [junit4]   2> 2244243 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2244497 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4370e392{/,null,AVAILABLE}
   [junit4]   2> 2245976 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@311e8486{SSL,[ssl, http/1.1]}{127.0.0.1:39353}
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server Started @2263827ms
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=39353, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/cores}
   [junit4]   2> 2245977 ERROR (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2245977 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-10T06:15:04.448Z
   [junit4]   2> 2246073 INFO  (zkConnectionManagerCallback-678-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2246113 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2246113 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/solr.xml
   [junit4]   2> 2246116 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2246116 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2246117 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2246585 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43531/solr
   [junit4]   2> 2246788 INFO  (zkConnectionManagerCallback-682-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2247031 INFO  (zkConnectionManagerCallback-684-thread-1-processing-n:127.0.0.1:39353_) [n:127.0.0.1:39353_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2247066 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2247067 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2247086 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39353_
   [junit4]   2> 2247105 INFO  (zkCallback-675-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2247105 INFO  (zkCallback-665-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2247453 INFO  (zkCallback-683-thread-1-processing-n:127.0.0.1:39353_) [n:127.0.0.1:39353_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2248384 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2248465 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2248465 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2248529 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:39353_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/cores
   [junit4]   2> 2248760 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:39353_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2248934 INFO  (OverseerThreadFactory-2045-thread-3-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:39353_ for creating new replica
   [junit4]   2> 2249402 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2249402 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 2250557 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 2250700 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema [collection1_shard1_replica_n21] Schema name=test
   [junit4]   2> 2253898 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2254287 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n21' using configuration from collection collection1, trusted=true
   [junit4]   2> 2254287 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n21' (registry 'solr.core.collection1.shard1.replica_n21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f
   [junit4]   2> 2254287 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2254287 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [[collection1_shard1_replica_n21] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/cores/collection1_shard1_replica_n21], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-1-001/cores/collection1_shard1_replica_n21/data/]
   [junit4]   2> 2254408 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=88.177734375, floorSegmentMB=1.3203125, forceMergeDeletesPctAllowed=26.2332621940895, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2254455 WARN  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2255465 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2255465 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2255466 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2255466 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2255480 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2255490 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.SolrIndexSearcher Opening [Searcher@370f0a01[collection1_shard1_replica_n21] main]
   [junit4]   2> 2255511 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2255512 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2255514 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 2255566 INFO  (searcherExecutor-2059-thread-1-processing-n:127.0.0.1:39353_ x:collection1_shard1_replica_n21 s:shard1 c:collection1) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [collection1_shard1_replica_n21] Registered new searcher Searcher@370f0a01[collection1_shard1_replica_n21] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2255566 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586376489342861312
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:39353/collection1_shard1_replica_n21/
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy https://127.0.0.1:39353/collection1_shard1_replica_n21/ has no replicas
   [junit4]   2> 2255595 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 2255616 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:39353/collection1_shard1_replica_n21/ shard1
   [junit4]   2> 2255617 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2255635 INFO  (qtp1320399144-3913) [n:127.0.0.1:39353_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=6232
   [junit4]   2> 2255703 INFO  (qtp1677638809-3863) [n:127.0.0.1:37799_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:39353_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=6943
   [junit4]   2> 2257168 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-2-001 of type NRT
   [junit4]   2> 2257206 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 2257210 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6b954e9a{/,null,AVAILABLE}
   [junit4]   2> 2257211 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@fd85aeb{SSL,[ssl, http/1.1]}{127.0.0.1:60432}
   [junit4]   2> 2257211 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.e.j.s.Server Started @2275061ms
   [junit4]   2> 2257211 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=60432, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-2-001/cores}
   [junit4]   2> 2257211 ERROR (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2257258 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 2257258 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2257258 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2257258 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-10T06:15:15.729Z
   [junit4]   2> 2257572 INFO  (zkConnectionManagerCallback-687-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2257589 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2257589 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.TestCryptoKeys_5DDE6DD7E5FA5A2A-001/shard-2-001/solr.xml
   [junit4]   2> 2257592 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 2257592 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 2257610 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@78aa3e7f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2257614 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43531/solr
   [junit4]   2> 2257823 INFO  (OverseerCollectionConfigSetProcessor-99148528425697285-127.0.0.1:37799_-n_0000000000) [n:127.0.0.1:37799_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2257838 INFO  (zkConnectionManagerCallback-691-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2258121 INFO  (zkConnectionManagerCallback-693-thread-1-processing-n:127.0.0.1:60432_) [n:127.0.0.1:60432_    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 2258282 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:60432_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2258283 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:60432_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2258293 INFO  (TEST-TestCryptoKeys.test-seed#[5DDE6DD7E5FA5A2A]) [n:127.0.0.1:60432_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60432_
   [junit4]   2> 2258293 INFO  (zkCallback-683-thread-1-processing-n:127.0.0.1:39353_) [n:127.0.0.1:39353_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2258294 INFO  (zkCallback-675-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2258294 INFO  (zkCallback-665-thread-2-processing-n:127.0.0.1:37799_) [n:127.0.0.1:37799_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2258434 INFO  (zkCallback-692-thread-1-processing-n:127.0.0.1:60432_) [n:127.0.0.1:

[...truncated too long message...]

OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 9561415 INFO  (jetty-closer-3827-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@ce20312{/solr,null,UNAVAILABLE}
   [junit4]   2> 9561454 INFO  (jetty-closer-3827-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4b1b1608{/solr,null,UNAVAILABLE}
   [junit4]   2> 9561508 INFO  (zkCallback-3858-thread-7-processing-n:127.0.0.1:49189_solr) [n:127.0.0.1:49189_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:49189_solr
   [junit4]   2> 9561527 INFO  (zkCallback-3864-thread-10-processing-n:127.0.0.1:34960_solr) [n:127.0.0.1:34960_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34960_solr
   [junit4]   2> 9563013 INFO  (jetty-closer-3827-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@71d6d3ad{/solr,null,UNAVAILABLE}
   [junit4]   2> 9563036 INFO  (jetty-closer-3827-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5ba4cdfa{/solr,null,UNAVAILABLE}
   [junit4]   2> 9563051 ERROR (SUITE-TestTolerantUpdateProcessorCloud-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 9563053 INFO  (SUITE-TestTolerantUpdateProcessorCloud-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:57364 57364
   [junit4]   2> 9563171 INFO  (Thread-4195) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:57364 57364
   [junit4]   2> 9563177 WARN  (Thread-4195) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	12	/solr/aliases.json
   [junit4]   2> 	10	/solr/security.json
   [junit4]   2> 	4	/solr/configs/solrCloudCollectionConfig/managed-schema
   [junit4]   2> 	4	/solr/configs/solrCloudCollectionConfig
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	20	/solr/collections/test_col/state.json
   [junit4]   2> 	12	/solr/clusterprops.json
   [junit4]   2> 	12	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/autoscaling.json
   [junit4]   2> 	2	/solr/overseer_elect/election/99148996429938706-127.0.0.1:49189_solr-n_0000000006
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	12	/solr/live_nodes
   [junit4]   2> 	12	/solr/collections
   [junit4]   2> 	2	/solr/overseer/queue
   [junit4]   2> 	2	/solr/autoscaling/events/.auto_add_replicas
   [junit4]   2> 	2	/solr/overseer/collection-queue-work
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 9563208 INFO  (SUITE-TestTolerantUpdateProcessorCloud-seed#[5DDE6DD7E5FA5A2A]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Dec 10, 2017 8:17:04 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=240, maxMBSortInHeap=5.241533852929293, sim=RandomSimilarity(queryNorm=false): {}, locale=mk-MK, timezone=SystemV/EST5
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=306496192,total=525336576
   [junit4]   2> NOTE: All tests run in this JVM: [LukeRequestHandlerTest, SolrIndexSplitterTest, CdcrReplicationHandlerTest, TestStressLucene, TestNestedDocsSort, TestSchemaManager, BlockCacheTest, TestRestoreCore, ClassificationUpdateProcessorIntegrationTest, TlogReplayBufferedWhileIndexingTest, SynonymTokenizerTest, CoreAdminCreateDiscoverTest, TestDocumentBuilder, AutoScalingHandlerTest, SampleTest, MoveReplicaHDFSFailoverTest, JavabinLoaderTest, RegexBoostProcessorTest, TestReplicaProperties, TestFieldCollectionResource, TestJettySolrRunner, TestStressUserVersions, TestEmbeddedSolrServerAdminHandler, BlockJoinFacetSimpleTest, AddReplicaTest, DirectUpdateHandlerTest, HighlighterConfigTest, SolrMetricsIntegrationTest, TestLazyCores, SignatureUpdateProcessorFactoryTest, TestLuceneMatchVersion, DeleteShardTest, TestApiFramework, TestHdfsBackupRestoreCore, DocumentAnalysisRequestHandlerTest, DOMUtilTest, TestFieldCache, TestLeaderElectionZkExpiry, SolrCoreMetricManagerTest, ShardRoutingCustomTest, QueryEqualityTest, SOLR749Test, TestPrepRecovery, SchemaVersionSpecificBehaviorTest, TestSolrCoreSnapshots, DocValuesMissingTest, PrimUtilsTest, TestClassicSimilarityFactory, ChaosMonkeySafeLeaderTest, InfoHandlerTest, DistribDocExpirationUpdateProcessorTest, TestJsonRequest, TestImpersonationWithHadoopAuth, TestCollapseQParserPlugin, HttpSolrCallGetCoreTest, DistributedTermsComponentTest, TestStressRecovery, HdfsDirectoryTest, TestSSLRandomization, OpenExchangeRatesOrgProviderTest, TestHdfsUpdateLog, MBeansHandlerTest, LegacyCloudClusterPropTest, TestSolrCLIRunExample, TestAuthenticationFramework, TestCustomSort, TestQuerySenderNoQuery, PeerSyncReplicationTest, TestPseudoReturnFields, TestCustomDocTransformer, TestUniqueKeyFieldResource, ConcurrentDeleteAndCreateCollectionTest, ImplicitSnitchTest, AtomicUpdateProcessorFactoryTest, TestXmlQParserPlugin, SuggesterTSTTest, PreAnalyzedFieldTest, TestRawResponseWriter, TestSolrCloudWithKerberosAlt, TestNumericRangeQuery32, NumericFieldsTest, TestSizeLimitedDistributedMap, SolrIndexMetricsTest, DistributedIntervalFacetingTest, RequestHandlersTest, TestSurroundQueryParser, TestLegacyFieldCache, TestSimpleTrackingShardHandler, TestBM25SimilarityFactory, TestGraphMLResponseWriter, CreateCollectionCleanupTest, HdfsBasicDistributedZk2Test, TestAnalyzedSuggestions, TestFieldTypeResource, TestBlobHandler, PingRequestHandlerTest, HdfsSyncSliceTest, MultiThreadedOCPTest, TestStressReorder, TestRemoteStreaming, TestStressInPlaceUpdates, SolrCmdDistributorTest, SpellingQueryConverterTest, TestSubQueryTransformerCrossCore, TestLRUStatsCache, SpellCheckCollatorWithCollapseTest, RankQueryTest, SolrCLIZkUtilsTest, TestMaxTokenLenTokenizer, CloudExitableDirectoryReaderTest, OutputWriterTest, CursorMarkTest, TestWriterPerf, SolrRequestParserTest, HdfsAutoAddReplicasIntegrationTest, TestSlowCompositeReaderWrapper, ZkSolrClientTest, BinaryUpdateRequestHandlerTest, TestCursorMarkWithoutUniqueKey, DocValuesMultiTest, LeaderFailureAfterFreshStartTest, CdcrBidirectionalTest, TestUseDocValuesAsStored2, BasicDistributedZkTest, TestCSVLoader, TestAtomicUpdateErrorCases, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestQueryUtils, TestOmitPositions, XmlUpdateRequestHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, TestCollationField, ReturnFieldsTest, TestPHPSerializedResponseWriter, UpdateParamsTest, ResponseLogComponentTest, TestDocSet, TestBinaryField, TestElisionMultitermQuery, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, EchoParamsTest, TestLMDirichletSimilarityFactory, TestFastOutputStream, PluginInfoTest, TestFastLRUCache, DateMathParserTest, TestSolrJ, ZkNodePropsTest, UUIDFieldTest, CircularListTest, TestRTGBase, TestEmbeddedSolrServerSchemaAPI, ConnectionReuseTest, AssignTest, LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, MoveReplicaTest, ReplicationFactorTest, RollingRestartTest, SolrXmlInZkTest, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestCollectionAPI, TestConfigSetsAPI, TestConfigSetsAPIExclusivity, TestDeleteCollectionOnDownNodes, TestDownShardTolerantSearch, TestExclusionRuleCollectionAccess, TestRandomRequestDistribution, TestRequestForwarding, TestRequestStatusCollectionAPI, TestSegmentSorting, TestTolerantUpdateProcessorCloud]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestTolerantUpdateProcessorCloud -Dtests.seed=5DDE6DD7E5FA5A2A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=mk-MK -Dtests.timezone=SystemV/EST5 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestTolerantUpdateProcessorCloud (suite) <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:34960/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000000
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5DDE6DD7E5FA5A2A]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > 	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorCloud.java:121)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [559/751 (6!)] on J1 in 203.45s, 0 tests, 1 error <<< FAILURES!

[...truncated 48146 lines...]

[JENKINS] Lucene-Solr-Tests-7.x - Build # 278 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/278/

3 tests failed.
FAILED:  org.apache.solr.cloud.DistributedQueueTest.testPeekElements

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([91D5E80E9902C4B:F433E4A139A97856]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.DistributedQueueTest.testPeekElements(DistributedQueueTest.java:265)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.MoveReplicaHDFSTest.test

Error Message:
Error from server at https://127.0.0.1:33985/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000005

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:33985/solr: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000005
	at __randomizedtesting.SeedInfo.seed([91D5E80E9902C4B:8149615A476C41B3]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:444)
	at org.apache.solr.cloud.MoveReplicaTest.beforeTest(MoveReplicaTest.java:95)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelCommitStream

Error Message:
expected:<5> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<5> but was:<2>
	at __randomizedtesting.SeedInfo.seed([72515E0A109A540E:52BB3C0A8CDBB942]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelCommitStream(StreamExpressionTest.java:4594)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11771 lines...]
   [junit4] Suite: org.apache.solr.cloud.DistributedQueueTest
   [junit4]   2> 663149 INFO  (SUITE-DistributedQueueTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.DistributedQueueTest_91D5E80E9902C4B-001/init-core-data-001
   [junit4]   2> 663150 WARN  (SUITE-DistributedQueueTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2171 numCloses=2171
   [junit4]   2> 663150 INFO  (SUITE-DistributedQueueTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 663151 INFO  (SUITE-DistributedQueueTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 663212 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLeakChildWatcher
   [junit4]   2> 663212 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 663212 INFO  (Thread-84) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 663212 INFO  (Thread-84) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 663466 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:40620
   [junit4]   2> 663483 ERROR (Thread-84) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 663637 INFO  (zkConnectionManagerCallback-85-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 663925 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testLeakChildWatcher
   [junit4]   2> 663932 ERROR (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 663932 INFO  (TEST-DistributedQueueTest.testLeakChildWatcher-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40620 40620
   [junit4]   2> 664203 INFO  (Thread-84) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40620 40620
   [junit4]   2> 664324 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPeekElements
   [junit4]   2> 664324 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 664362 INFO  (Thread-85) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 664362 INFO  (Thread-85) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 664439 ERROR (Thread-85) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 664470 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:55309
   [junit4]   2> 664606 INFO  (zkConnectionManagerCallback-88-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 666729 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testPeekElements
   [junit4]   2> 666733 ERROR (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 666733 INFO  (TEST-DistributedQueueTest.testPeekElements-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55309 55309
   [junit4]   2> 666919 INFO  (Thread-85) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:55309 55309
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistributedQueueTest -Dtests.method=testPeekElements -Dtests.seed=91D5E80E9902C4B -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar-BH -Dtests.timezone=NST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 2.72s J0 | DistributedQueueTest.testPeekElements <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([91D5E80E9902C4B:F433E4A139A97856]:0)
   [junit4]    > 	at org.apache.solr.cloud.DistributedQueueTest.testPeekElements(DistributedQueueTest.java:265)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 666965 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueue
   [junit4]   2> 666966 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 666966 INFO  (Thread-86) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 666966 INFO  (Thread-86) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 667626 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:51431
   [junit4]   2> 667766 ERROR (Thread-86) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 667855 INFO  (zkConnectionManagerCallback-91-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 670101 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueue
   [junit4]   2> 670139 ERROR (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 670139 INFO  (TEST-DistributedQueueTest.testDistributedQueue-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51431 51431
   [junit4]   2> 670475 INFO  (Thread-86) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51431 51431
   [junit4]   2> 670477 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueueCache
   [junit4]   2> 670477 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 670482 INFO  (Thread-89) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 670482 INFO  (Thread-89) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 670586 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:42368
   [junit4]   2> 671036 ERROR (Thread-89) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 672158 INFO  (zkConnectionManagerCallback-94-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 672190 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueueCache
   [junit4]   2> 672202 ERROR (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 672203 INFO  (TEST-DistributedQueueTest.testDistributedQueueCache-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42368 42368
   [junit4]   2> 672439 INFO  (Thread-89) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42368 42368
   [junit4]   2> 672454 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueueBlocking
   [junit4]   2> 672455 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 672498 INFO  (Thread-90) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 672499 INFO  (Thread-90) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 672500 ERROR (Thread-90) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 672618 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:54368
   [junit4]   2> 672678 INFO  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 674118 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46e15243 name: ZooKeeperConnection Watcher:127.0.0.1:54368/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 674118 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 676159 WARN  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]-SendThread(127.0.0.1:54368)) [    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x1603c0856050000 has expired
   [junit4]   2> 676159 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46e15243 name: ZooKeeperConnection Watcher:127.0.0.1:54368/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 676159 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 676166 WARN  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 676271 INFO  (zkConnectionManagerCallback-97-thread-1-EventThread) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 676271 INFO  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 676271 INFO  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 676271 INFO  (zkConnectionManagerCallback-97-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient Connected:true
   [junit4]   2> 677438 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueueBlocking
   [junit4]   2> 677447 ERROR (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 677447 INFO  (TEST-DistributedQueueTest.testDistributedQueueBlocking-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54368 54368
   [junit4]   2> 678114 INFO  (Thread-90) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54368 54368
   [junit4]   2> 678115 WARN  (Thread-90) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	2	/solr/distqueue/test
   [junit4]   2> 
   [junit4]   2> 678227 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLocallyOffer
   [junit4]   2> 678228 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 678273 INFO  (Thread-91) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 678273 INFO  (Thread-91) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 678326 ERROR (Thread-91) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 678412 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer start zk server on port:37747
   [junit4]   2> 678978 INFO  (zkConnectionManagerCallback-100-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 681873 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testLocallyOffer
   [junit4]   2> 681891 ERROR (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 681891 INFO  (TEST-DistributedQueueTest.testLocallyOffer-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37747 37747
   [junit4]   2> 682275 INFO  (Thread-91) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37747 37747
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J0/temp/solr.cloud.DistributedQueueTest_91D5E80E9902C4B-001
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-BH, timezone=NST
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=94191480,total=305659904
   [junit4]   2> NOTE: All tests run in this JVM: [UpdateParamsTest, TestFilteredDocIdSet, QueryParsingTest, RulesTest, TestJsonFacets, DistributedQueueTest]
   [junit4] Completed [32/751 (1!)] on J0 in 22.08s, 6 tests, 1 failure <<< FAILURES!

[...truncated 1792 lines...]
   [junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
   [junit4]   2> 7515951 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/init-core-data-001
   [junit4]   2> 7515951 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
   [junit4]   2> 7515952 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 7515953 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 7515954 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001
   [junit4]   2> 7515954 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 7515954 INFO  (Thread-3615) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 7515954 INFO  (Thread-3615) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 7515955 ERROR (Thread-3615) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 7516057 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:45661
   [junit4]   2> 7516063 INFO  (zkConnectionManagerCallback-3632-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516093 INFO  (jetty-launcher-3629-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7516094 INFO  (jetty-launcher-3629-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7516098 INFO  (jetty-launcher-3629-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7516103 INFO  (jetty-launcher-3629-thread-4) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7516111 INFO  (jetty-launcher-3629-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@41c5a387{/solr,null,AVAILABLE}
   [junit4]   2> 7516111 INFO  (jetty-launcher-3629-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@20c6fff2{SSL,[ssl, http/1.1]}{127.0.0.1:36881}
   [junit4]   2> 7516111 INFO  (jetty-launcher-3629-thread-3) [    ] o.e.j.s.Server Started @7533075ms
   [junit4]   2> 7516111 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36881}
   [junit4]   2> 7516112 ERROR (jetty-launcher-3629-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7516112 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 7516112 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7516112 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7516112 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:01:49.822Z
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@76baca41{/solr,null,AVAILABLE}
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@16ae16c1{SSL,[ssl, http/1.1]}{127.0.0.1:52835}
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.e.j.s.Server Started @7533078ms
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52835}
   [junit4]   2> 7516114 ERROR (jetty-launcher-3629-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7516114 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:01:49.824Z
   [junit4]   2> 7516134 INFO  (jetty-launcher-3629-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@cff8903{/solr,null,AVAILABLE}
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1c771c69{SSL,[ssl, http/1.1]}{127.0.0.1:48847}
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.e.j.s.Server Started @7533099ms
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=48847}
   [junit4]   2> 7516135 ERROR (jetty-launcher-3629-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7516135 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:01:49.845Z
   [junit4]   2> 7516215 INFO  (jetty-launcher-3629-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7eb59e2{/solr,null,AVAILABLE}
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@74e9fb0e{SSL,[ssl, http/1.1]}{127.0.0.1:55369}
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.e.j.s.Server Started @7533180ms
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55369}
   [junit4]   2> 7516216 ERROR (jetty-launcher-3629-thread-4) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7516216 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:01:49.926Z
   [junit4]   2> 7516247 INFO  (zkConnectionManagerCallback-3634-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516247 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7516316 INFO  (zkConnectionManagerCallback-3638-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516327 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7516339 INFO  (zkConnectionManagerCallback-3636-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516340 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7516454 INFO  (jetty-launcher-3629-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7516455 INFO  (zkConnectionManagerCallback-3640-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516482 INFO  (jetty-launcher-3629-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7516482 INFO  (jetty-launcher-3629-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7516483 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7516590 INFO  (zkConnectionManagerCallback-3652-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516577 INFO  (zkConnectionManagerCallback-3644-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516577 INFO  (zkConnectionManagerCallback-3650-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516738 INFO  (jetty-launcher-3629-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7516758 INFO  (zkConnectionManagerCallback-3656-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516806 INFO  (zkConnectionManagerCallback-3660-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516810 INFO  (zkConnectionManagerCallback-3658-thread-1-processing-n:127.0.0.1:48847_solr) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516811 INFO  (zkConnectionManagerCallback-3662-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516906 INFO  (zkConnectionManagerCallback-3666-thread-1-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7516992 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7516992 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55369_solr
   [junit4]   2> 7516994 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.Overseer Overseer (id=99145646713405452-127.0.0.1:55369_solr-n_0000000000) starting
   [junit4]   2> 7516995 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7516996 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36881_solr
   [junit4]   2> 7517018 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 7517055 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55369_solr
   [junit4]   2> 7517055 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 7517114 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 7517127 INFO  (zkCallback-3665-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 7517246 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 7517249 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7517311 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 7517312 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7517331 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48847_solr
   [junit4]   2> 7517334 INFO  (zkCallback-3665-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 7517380 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 7517387 INFO  (zkCallback-3657-thread-1-processing-n:127.0.0.1:48847_solr) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 7517387 INFO  (zkCallback-3661-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 7517398 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52835_solr
   [junit4]   2> 7517510 INFO  (zkCallback-3657-thread-1-processing-n:127.0.0.1:48847_solr) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 7517510 INFO  (zkCallback-3661-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 7517510 INFO  (zkCallback-3665-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 7517598 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 7519731 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36881.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519783 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36881.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519783 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36881.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519784 INFO  (jetty-launcher-3629-thread-3) [n:127.0.0.1:36881_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node3/.
   [junit4]   2> 7519896 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55369.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519964 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55369.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519964 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55369.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7519965 INFO  (jetty-launcher-3629-thread-4) [n:127.0.0.1:55369_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node4/.
   [junit4]   2> 7520008 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52835.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520010 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48847.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520319 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52835.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520320 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52835.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520320 INFO  (jetty-launcher-3629-thread-2) [n:127.0.0.1:52835_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node2/.
   [junit4]   2> 7520324 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48847.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520324 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48847.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7520325 INFO  (jetty-launcher-3629-thread-1) [n:127.0.0.1:48847_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node1/.
   [junit4]   2> 7520731 INFO  (zkConnectionManagerCallback-3671-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7520734 INFO  (zkConnectionManagerCallback-3675-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7520734 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 7520735 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45661/solr ready
   [junit4]   2> 7520907 INFO  (qtp1026759654-17396) [n:127.0.0.1:55369_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 7521014 INFO  (qtp1026759654-17396) [n:127.0.0.1:55369_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=107
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 7521961 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 7522102 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 7522104 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 7522235 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_43029_hdfs____.bh5gam/webapp
   [junit4]   2> 7523053 INFO  (OverseerCollectionConfigSetProcessor-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 7524743 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43029
   [junit4]   2> 7529954 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 7529955 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 7530342 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46013_datanode____alc31z/webapp
   [junit4]   2> 7533637 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46013
   [junit4]   2> 7534968 WARN  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 7535019 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 7535180 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_36421_datanode____i93l7b/webapp
   [junit4]   2> 7536491 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-002/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:40621) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 7536714 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x2e85555a10eb1a: from storage DS-4c393cb1-2143-4f8b-8754-cc7e80df835e node DatanodeRegistration(127.0.0.1:50668, datanodeUuid=c91443d4-2152-4d5c-a5a2-1239693cb7a8, infoPort=57431, infoSecurePort=0, ipcPort=34522, storageInfo=lv=-56;cid=testClusterID;nsid=558505609;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 7536714 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x2e85555a10eb1a: from storage DS-e4d231ea-d4ed-414a-95f9-fc7f1510d53a node DatanodeRegistration(127.0.0.1:50668, datanodeUuid=c91443d4-2152-4d5c-a5a2-1239693cb7a8, infoPort=57431, infoSecurePort=0, ipcPort=34522, storageInfo=lv=-56;cid=testClusterID;nsid=558505609;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 7537916 INFO  (SUITE-MoveReplicaHDFSTest-seed#[91D5E80E9902C4B]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36421
   [junit4]   2> 7539143 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-002/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:40621) [    ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
   [junit4]   2> 7539301 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x2e8555ec8dc4a4: from storage DS-08a50d7e-6aa0-475a-bfb5-c65329252d52 node DatanodeRegistration(127.0.0.1:41673, datanodeUuid=0f9bea97-fc0c-4fd5-82b0-29b83f87a9bc, infoPort=56167, infoSecurePort=0, ipcPort=37351, storageInfo=lv=-56;cid=testClusterID;nsid=558505609;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 7539302 INFO  (Block report processor) [    ] BlockStateChange BLOCK* processReport 0x2e8555ec8dc4a4: from storage DS-3996a272-06ee-47ec-804c-a1e76c369319 node DatanodeRegistration(127.0.0.1:41673, datanodeUuid=0f9bea97-fc0c-4fd5-82b0-29b83f87a9bc, infoPort=56167, infoSecurePort=0, ipcPort=37351, storageInfo=lv=-56;cid=testClusterID;nsid=558505609;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 7540799 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testFailedMove
   [junit4]   2> 7540800 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 7540801 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 7540946 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e35d4cf{/solr,null,AVAILABLE}
   [junit4]   2> 7540947 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@71f127a7{SSL,[ssl, http/1.1]}{127.0.0.1:33985}
   [junit4]   2> 7540947 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.e.j.s.Server Started @7557911ms
   [junit4]   2> 7540947 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33985}
   [junit4]   2> 7540947 ERROR (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 7540947 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.3.0
   [junit4]   2> 7540948 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 7540948 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 7540948 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-09T18:02:14.658Z
   [junit4]   2> 7540974 INFO  (zkConnectionManagerCallback-3678-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7540975 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 7541032 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45661/solr
   [junit4]   2> 7541051 INFO  (zkConnectionManagerCallback-3682-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7541155 INFO  (zkConnectionManagerCallback-3684-thread-1-processing-n:127.0.0.1:33985_solr) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7541182 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 7541203 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 7541218 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33985_solr
   [junit4]   2> 7541220 INFO  (zkCallback-3674-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541235 INFO  (zkCallback-3657-thread-1-processing-n:127.0.0.1:48847_solr) [n:127.0.0.1:48847_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541235 INFO  (zkCallback-3661-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541235 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541279 INFO  (zkCallback-3674-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541279 INFO  (zkCallback-3665-thread-1-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7541294 INFO  (zkCallback-3683-thread-1-processing-n:127.0.0.1:33985_solr) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 7542349 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33985.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7542391 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33985.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7542391 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33985.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7542393 INFO  (TEST-MoveReplicaHDFSTest.testFailedMove-seed#[91D5E80E9902C4B]) [n:127.0.0.1:33985_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node5/.
   [junit4]   2> 7543358 INFO  (zkConnectionManagerCallback-3687-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 7543613 INFO  (qtp701827325-17637) [n:127.0.0.1:33985_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&autoAddReplicas=false&name=MoveReplicaHDFSTest_failed_coll_true&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 7543646 INFO  (OverseerThreadFactory-7718-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.CreateCollectionCmd Create collection MoveReplicaHDFSTest_failed_coll_true
   [junit4]   2> 7543904 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:33985/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 7543971 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:55369/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 7543983 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:52835/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 7543998 INFO  (OverseerStateUpdate-99145646713405452-127.0.0.1:55369_solr-n_0000000000) [n:127.0.0.1:55369_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"MoveReplicaHDFSTest_failed_coll_true",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:36881/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 7544518 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 7544518 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 7544519 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 7544520 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 7544563 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 7544563 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 7544563 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf1&newCollection=true&name=MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7&action=CREATE&numShards=2&collection=MoveReplicaHDFSTest_failed_coll_true&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 7544563 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 7544679 INFO  (zkCallback-3661-thread-1-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544679 INFO  (zkCallback-3665-thread-1-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544679 INFO  (zkCallback-3665-thread-2-processing-n:127.0.0.1:55369_solr) [n:127.0.0.1:55369_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544682 INFO  (zkCallback-3659-thread-1-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544682 INFO  (zkCallback-3683-thread-1-processing-n:127.0.0.1:33985_solr) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544683 INFO  (zkCallback-3661-thread-2-processing-n:127.0.0.1:52835_solr) [n:127.0.0.1:52835_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544683 INFO  (zkCallback-3683-thread-2-processing-n:127.0.0.1:33985_solr) [n:127.0.0.1:33985_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7544684 INFO  (zkCallback-3659-thread-2-processing-n:127.0.0.1:36881_solr) [n:127.0.0.1:36881_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MoveReplicaHDFSTest_failed_coll_true/state.json] for collection [MoveReplicaHDFSTest_failed_coll_true] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 7545599 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 7545615 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 7545900 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 7545904 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] Schema name=minimal
   [junit4]   2> 7545906 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 7545906 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 7545907 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55369.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n3' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7545936 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 7548091 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 7548092 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 7548092 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33985.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n1' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7548100 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] Schema name=minimal
   [junit4]   2> 7548102 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 7548102 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 7548111 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40621/data
   [junit4]   2> 7548112 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 7548112 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 7548112 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node4/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3], dataDir=[hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node4/data/]
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36881.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n7' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n7') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40621/data
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 7548159 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node3/MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7], dataDir=[hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data/]
   [junit4]   2> 7548161 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data/snapshot_metadata
   [junit4]   2> 7548114 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40621/data
   [junit4]   2> 7548178 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 7548178 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 7548178 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node5/MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1], dataDir=[hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node2/data/]
   [junit4]   2> 7548179 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node2/data/snapshot_metadata
   [junit4]   2> 7548308 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node4/data/snapshot_metadata
   [junit4]   2> 7548315 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 7549318 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.s.IndexSchema [MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] Schema name=minimal
   [junit4]   2> 7549321 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 7549321 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5' using configuration from collection MoveReplicaHDFSTest_failed_coll_true, trusted=true
   [junit4]   2> 7549321 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52835.solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n5' (registry 'solr.core.MoveReplicaHDFSTest_failed_coll_true.shard2.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@31748e1a
   [junit4]   2> 7549322 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:40621/data
   [junit4]   2> 7549322 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 7549322 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 7549322 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.SolrCore [[MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSTest_91D5E80E9902C4B-001/tempDir-001/node2/MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5], dataDir=[hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node6/data/]
   [junit4]   2> 7549448 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data
   [junit4]   2> 7549467 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node6/data/snapshot_metadata
   [junit4]   2> 7549638 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node2/data
   [junit4]   2> 7549654 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node4/data
   [junit4]   2> 7549828 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node6/data
   [junit4]   2> 7549968 INFO  (qtp701827325-17634) [n:127.0.0.1:33985_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node2 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node2/data/index
   [junit4]   2> 7550034 INFO  (qtp704984563-17362) [n:127.0.0.1:36881_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node8 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n7] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node8/data/index
   [junit4]   2> 7550055 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node4/data/index
   [junit4]   2> 7550402 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:40621/data/MoveReplicaHDFSTest_failed_coll_true/core_node6/data/index
   [junit4]   2> 7552937 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50668 is added to blk_1073741827_1003{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c393cb1-2143-4f8b-8754-cc7e80df835e:NORMAL:127.0.0.1:50668|RBW], ReplicaUC[[DISK]DS-3996a272-06ee-47ec-804c-a1e76c369319:NORMAL:127.0.0.1:41673|RBW]]} size 69
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50668 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-08a50d7e-6aa0-475a-bfb5-c65329252d52:NORMAL:127.0.0.1:41673|RBW], ReplicaUC[[DISK]DS-4c393cb1-2143-4f8b-8754-cc7e80df835e:NORMAL:127.0.0.1:50668|RBW]]} size 0
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50668 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-08a50d7e-6aa0-475a-bfb5-c65329252d52:NORMAL:127.0.0.1:41673|RBW], ReplicaUC[[DISK]DS-e4d231ea-d4ed-414a-95f9-fc7f1510d53a:NORMAL:127.0.0.1:50668|RBW]]} size 69
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50668 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-3996a272-06ee-47ec-804c-a1e76c369319:NORMAL:127.0.0.1:41673|RBW], ReplicaUC[[DISK]DS-e4d231ea-d4ed-414a-95f9-fc7f1510d53a:NORMAL:127.0.0.1:50668|RBW]]} size 0
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41673 is added to blk_1073741827_1003 size 69
   [junit4]   2> 7552940 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41673 is added to blk_1073741825_1001 size 69
   [junit4]   2> 7553154 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41673 is added to blk_1073741828_1004 size 69
   [junit4]   2> 7553155 INFO  (Block report processor) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41673 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e4d231ea-d4ed-414a-95f9-fc7f1510d53a:NORMAL:127.0.0.1:50668|RBW], ReplicaUC[[DISK]DS-08a50d7e-6aa0-475a-bfb5-c65329252d52:NORMAL:127.0.0.1:41673|RBW]]} size 0
   [junit4]   2> 7554056 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 7554056 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 7554056 INFO  (qtp1026759654-17398) [n:127.0.0.1:55369_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard1 r:core_node4 x:MoveReplicaHDFSTest_failed_coll_true_shard1_replica_n3] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 7554157 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 7554157 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:core_node6 x:MoveReplicaHDFSTest_failed_coll_true_shard2_replica_n5] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 7554157 INFO  (qtp154162816-17373) [n:127.0.0.1:52835_solr c:MoveReplicaHDFSTest_failed_coll_true s:shard2 r:

[...truncated too long message...]

ted live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 1020650 INFO  (zkCallback-272-thread-5-processing-n:127.0.0.1:36238_solr) [n:127.0.0.1:36238_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 1020661 INFO  (jetty-closer-236-thread-4) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1020661 INFO  (jetty-closer-236-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1020661 INFO  (zkCallback-1805-thread-1-processing-n:127.0.0.1:37583_solr x:collection1_collection_shard2_replica_n2 s:shard2 c:collection1_collection r:core_node4) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 1020665 INFO  (coreCloseExecutor-576-thread-1) [n:127.0.0.1:36238_solr c:collection1_collection s:shard1 r:core_node3 x:collection1_collection_shard1_replica_n1] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@50e5f18b
   [junit4]   2> 1020665 INFO  (coreCloseExecutor-576-thread-1) [n:127.0.0.1:36238_solr c:collection1_collection s:shard1 r:core_node3 x:collection1_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1_collection.shard1.replica_n1, tag=1357246859
   [junit4]   2> 1020692 INFO  (coreCloseExecutor-576-thread-1) [n:127.0.0.1:36238_solr c:collection1_collection s:shard1 r:core_node3 x:collection1_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@51bdd4e8: rootName = solr_36238, domain = solr.core.collection1_collection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.collection1_collection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@7063501e
   [junit4]   2> 1020707 INFO  (coreCloseExecutor-577-thread-1) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@3f61282a
   [junit4]   2> 1020710 INFO  (coreCloseExecutor-576-thread-1) [n:127.0.0.1:36238_solr c:collection1_collection s:shard1 r:core_node3 x:collection1_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard1.leader, tag=1357246859
   [junit4]   2> 1020732 INFO  (coreCloseExecutor-577-thread-1) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1_collection.shard2.replica_n2, tag=1063331882
   [junit4]   2> 1020732 INFO  (coreCloseExecutor-577-thread-1) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@41dfc7bb: rootName = solr_37583, domain = solr.core.collection1_collection.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.collection1_collection.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@10d0f746
   [junit4]   2> 1020755 INFO  (coreCloseExecutor-577-thread-1) [n:127.0.0.1:37583_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard2.leader, tag=1063331882
   [junit4]   2> 1020784 INFO  (jetty-closer-236-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2a8bc0d6{/solr,null,UNAVAILABLE}
   [junit4]   2> 1020794 INFO  (jetty-closer-236-thread-1) [    ] o.a.s.c.Overseer Overseer (id=99145935423864841-127.0.0.1:37583_solr-n_0000000000) closing
   [junit4]   2> 1020794 INFO  (OverseerStateUpdate-99145935423864841-127.0.0.1:37583_solr-n_0000000000) [n:127.0.0.1:37583_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37583_solr
   [junit4]   2> 1020816 WARN  (OverseerAutoScalingTriggerThread-99145935423864841-127.0.0.1:37583_solr-n_0000000000) [n:127.0.0.1:37583_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1020844 INFO  (jetty-closer-236-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@12d318bc{/solr,null,UNAVAILABLE}
   [junit4]   2> 1020857 ERROR (SUITE-StreamExpressionTest-seed#[72515E0A109A540E]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1020858 INFO  (SUITE-StreamExpressionTest-seed#[72515E0A109A540E]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51520 51520
   [junit4]   2> 1021031 INFO  (Thread-196) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51520 51520
   [junit4]   2> 1021032 WARN  (Thread-196) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	79	/solr/aliases.json
   [junit4]   2> 	11	/solr/configs/conf
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	79	/solr/clusterprops.json
   [junit4]   2> 	79	/solr/clusterstate.json
   [junit4]   2> 	26	/solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 	22	/solr/collections/destinationCollection/state.json
   [junit4]   2> 	20	/solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 	17	/solr/collections/collection1_collection/state.json
   [junit4]   2> 	14	/solr/collections/mainCorpus1/state.json
   [junit4]   2> 	14	/solr/collections/destination1/state.json
   [junit4]   2> 	14	/solr/collections/workQueue/state.json
   [junit4]   2> 	12	/solr/collections/mainCorpus/state.json
   [junit4]   2> 	10	/solr/collections/destination/state.json
   [junit4]   2> 	10	/solr/collections/workQueue1/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/99145935423864841-127.0.0.1:37583_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	79	/solr/live_nodes
   [junit4]   2> 	79	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_72515E0A109A540E-001
   [junit4]   2> Dec 09, 2017 7:21:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {expr_s=FSTOrd50, join2_s=BlockTreeOrds(blocksize=128), multiDefault=BlockTreeOrds(blocksize=128), subject=BlockTreeOrds(blocksize=128), a1_s=Lucene50(blocksize=128), a2_s=FSTOrd50, test_t=Lucene50(blocksize=128), level1_s=FSTOrd50, ident_s=Lucene50(blocksize=128), body_t=Lucene50(blocksize=128), s_multi=FSTOrd50, a_ss=BlockTreeOrds(blocksize=128), col_s=Lucene50(blocksize=128), level2_s=Lucene50(blocksize=128), a_s=FSTOrd50, a_t=BlockTreeOrds(blocksize=128), side_s=Lucene50(blocksize=128), id=Lucene50(blocksize=128), text=FSTOrd50}, docValues:{expr_s=DocValuesFormat(name=Lucene70), join2_s=DocValuesFormat(name=Asserting), multiDefault=DocValuesFormat(name=Asserting), a1_s=DocValuesFormat(name=Lucene70), price_f=DocValuesFormat(name=Asserting), a_f=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Lucene70), join1_i=DocValuesFormat(name=Lucene70), a2_s=DocValuesFormat(name=Lucene70), a_i=DocValuesFormat(name=Memory), join3_i=DocValuesFormat(name=Memory), test_i=DocValuesFormat(name=Asserting), field_i=DocValuesFormat(name=Asserting), level1_s=DocValuesFormat(name=Lucene70), s_multi=DocValuesFormat(name=Lucene70), a_ss=DocValuesFormat(name=Asserting), col_s=DocValuesFormat(name=Lucene70), level2_s=DocValuesFormat(name=Lucene70), a_s=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Asserting), i_multi=DocValuesFormat(name=Memory), b_ls=DocValuesFormat(name=Lucene70), test_dt=DocValuesFormat(name=Lucene70), ident_s=DocValuesFormat(name=Lucene70), order_i=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Lucene70), side_s=DocValuesFormat(name=Memory), miles_i=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1578, maxMBSortInHeap=6.435731837867215, sim=RandomSimilarity(queryNorm=false): {}, locale=lt-LT, timezone=Etc/GMT+11
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=173714128,total=423624704
   [junit4]   2> NOTE: All tests run in this JVM: [TestSuggesterResponse, NormalizeEvaluatorTest, SelectWithEvaluatorsTest, PowerEvaluatorTest, HttpSolrClientBuilderTest, SolrExceptionTest, TestJsonRecordReader, TestCollectionAdminRequest, RegressionEvaluatorTest, MultiplyEvaluatorTest, JettyWebappTest, UuidEvaluatorTest, CloudSolrClientBuilderTest, NoOpResponseParserTest, TestCollectionStateWatchers, SolrParamTest, CommonParamsTest, SolrExampleXMLTest, SolrExampleBinaryTest, AddEvaluatorTest, LBHttpSolrClientBuilderTest, GreaterThanEqualToEvaluatorTest, LessThanEqualToEvaluatorTest, SolrExampleJettyTest, CoalesceEvaluatorTest, TestPolicy, CloudSolrClientCacheTest, CloudSolrClientMultiConstructorTest, CloudSolrClientTest, StreamExpressionTest]
   [junit4] Completed [146/146 (1!)] on J2 in 360.24s, 139 tests, 1 failure, 9 skipped <<< FAILURES!

[...truncated 46421 lines...]

[JENKINS] Lucene-Solr-Tests-7.x - Build # 277 - Still unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/277/

11 tests failed.
FAILED:  org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddTooManyReplicas

Error Message:
Could not load collection from ZK: TooManyReplicasInSeveralFlavors

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: TooManyReplicasInSeveralFlavors
	at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:60962F4041A265BE]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
	at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
	at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:110)
	at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
	at org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddTooManyReplicas(CollectionTooManyReplicasTest.java:91)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/TooManyReplicasInSeveralFlavors/state.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
	... 44 more


FAILED:  org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testRandom

Error Message:
Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason: <pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason:
<pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:9F887E97BE24EC7C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
	at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.assertFacetCountsAreCorrect(TestCloudJSONFacetJoinDomain.java:461)
	at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.assertFacetCountsAreCorrect(TestCloudJSONFacetJoinDomain.java:429)
	at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testRandom(TestCloudJSONFacetJoinDomain.java:370)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testBespoke

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection]
	at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:E61FDFA1C7C49DF8]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
	at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.assertFacetCountsAreCorrect(TestCloudJSONFacetJoinDomain.java:461)
	at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.assertFacetCountsAreCorrect(TestCloudJSONFacetJoinDomain.java:429)
	at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testBespoke(TestCloudJSONFacetJoinDomain.java:280)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason:
<pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	... 48 more


FAILED:  org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testMalformedGivesError

Error Message:
bogus -> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason: <pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html>  expected:<400> but was:<404>

Stack Trace:
java.lang.AssertionError: bogus -> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:47007/solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select. Reason:
<pre>    Can not find: /solr/org.apache.solr.cloud.TestCloudJSONFacetJoinDomain_collection/select</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
 expected:<400> but was:<404>
	at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:1A5BE24EBA1652FC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestCloudJSONFacetJoinDomain.testMalformedGivesError(TestCloudJSONFacetJoinDomain.java:211)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestHdfsCloudBackupRestore.test

Error Message:
expected:<COMPLETED> but was:<RUNNING>

Stack Trace:
java.lang.AssertionError: expected:<COMPLETED> but was:<RUNNING>
	at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:65906442A1B837F7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:287)
	at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:140)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestLeaderElectionWithEmptyReplica.test

Error Message:
Indexed documents not found expected:<10> but was:<0>

Stack Trace:
java.lang.AssertionError: Indexed documents not found expected:<10> but was:<0>
	at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:65906442A1B837F7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestLeaderElectionWithEmptyReplica.test(TestLeaderElectionWithEmptyReplica.java:105)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestSegmentSorting.testAtomicUpdateOfSegmentSortField

Error Message:
expected:<COMPLETED> but was:<RUNNING>

Stack Trace:
java.lang.AssertionError: expected:<COMPLETED> but was:<RUNNING>
	at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:CD67C9DD901C1B7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.cloud.TestSegmentSorting.createCollection(TestSegmentSorting.java:84)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:968)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TestPolicyCloud.testMetricsTag

Error Message:
KeeperErrorCode = Session expired for /autoscaling.json

Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json
	at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:669EBB32DC390D5F]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1327)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$setData$6(SolrZkClient.java:351)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:351)
	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:539)
	at org.apache.solr.cloud.autoscaling.TestPolicyCloud.after(TestPolicyCloud.java:79)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:53330_solr wasn't created

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:53330_solr wasn't created
	at __randomizedtesting.SeedInfo.seed([EDC45B980F445A0F:F57ED394017197E0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeMarkersRegistration(TriggerIntegrationTest.java:930)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest

Error Message:
Could not load collection from ZK: localShardsTestColl

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: localShardsTestColl
	at __randomizedtesting.SeedInfo.seed([8FC2650D0D48DB60:730FFDB4F39F4C9A]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1123)
	at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:648)
	at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:130)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
	at org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest(CloudSolrClientTest.java:429)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/localShardsTestColl/state.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
	at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
	at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1135)
	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1121)
	... 44 more


FAILED:  org.apache.solr.common.cloud.TestCollectionStateWatchers.testWatchesWorkForStateFormat1

Error Message:
CollectionStateWatcher not notified of stateformat=1 collection creation

Stack Trace:
java.lang.AssertionError: CollectionStateWatcher not notified of stateformat=1 collection creation
	at __randomizedtesting.SeedInfo.seed([8FC2650D0D48DB60:E80579E3F9F0FFC1]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.common.cloud.TestCollectionStateWatchers.testWatchesWorkForStateFormat1(TestCollectionStateWatchers.java:290)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11786 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionTooManyReplicasTest
   [junit4]   2> 840857 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/init-core-data-001
   [junit4]   2> 840858 WARN  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 841022 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 841023 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 841024 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001
   [junit4]   2> 841024 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 841026 INFO  (Thread-442) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 841026 INFO  (Thread-442) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 841079 ERROR (Thread-442) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 841126 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:40795
   [junit4]   2> 841232 INFO  (zkConnectionManagerCallback-282-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 841268 INFO  (jetty-launcher-279-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 843418 INFO  (jetty-launcher-279-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 843563 INFO  (jetty-launcher-279-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@81ed203{/solr,null,AVAILABLE}
   [junit4]   2> 843564 INFO  (jetty-launcher-279-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3c531598{SSL,[ssl, http/1.1]}{127.0.0.1:49728}
   [junit4]   2> 843564 INFO  (jetty-launcher-279-thread-2) [    ] o.e.j.s.Server Started @863517ms
   [junit4]   2> 843564 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49728}
   [junit4]   2> 843565 ERROR (jetty-launcher-279-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 843565 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 843565 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 843565 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 843565 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-08T21:13:01.975Z
   [junit4]   2> 843571 INFO  (jetty-launcher-279-thread-3) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4b312e2e{/solr,null,AVAILABLE}
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4008ff7e{SSL,[ssl, http/1.1]}{127.0.0.1:33000}
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.e.j.s.Server Started @863640ms
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33000}
   [junit4]   2> 843687 ERROR (jetty-launcher-279-thread-3) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 843687 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 843688 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 843688 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 843688 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-08T21:13:02.098Z
   [junit4]   2> 843871 INFO  (jetty-launcher-279-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@45d763ee{/solr,null,AVAILABLE}
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4c314cce{SSL,[ssl, http/1.1]}{127.0.0.1:56434}
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.e.j.s.Server Started @863825ms
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56434}
   [junit4]   2> 843872 ERROR (jetty-launcher-279-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.3.0
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 843872 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-12-08T21:13:02.282Z
   [junit4]   2> 843891 INFO  (zkConnectionManagerCallback-286-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844195 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 844219 INFO  (zkConnectionManagerCallback-284-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844274 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 844322 INFO  (jetty-launcher-279-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40795/solr
   [junit4]   2> 844414 INFO  (jetty-launcher-279-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40795/solr
   [junit4]   2> 844414 INFO  (zkConnectionManagerCallback-288-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844451 INFO  (zkConnectionManagerCallback-296-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844634 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 844713 INFO  (jetty-launcher-279-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40795/solr
   [junit4]   2> 844713 INFO  (zkConnectionManagerCallback-292-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844778 INFO  (zkConnectionManagerCallback-298-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 844779 INFO  (zkConnectionManagerCallback-302-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 845143 INFO  (zkConnectionManagerCallback-304-thread-1-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 845162 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 845166 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33000_solr
   [junit4]   2> 845182 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.Overseer Overseer (id=99140736085983240-127.0.0.1:33000_solr-n_0000000000) starting
   [junit4]   2> 845230 INFO  (zkConnectionManagerCallback-307-thread-1-processing-n:127.0.0.1:56434_solr) [n:127.0.0.1:56434_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 845290 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33000_solr
   [junit4]   2> 845354 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 845355 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56434_solr
   [junit4]   2> 845402 INFO  (zkCallback-303-thread-1-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 845557 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 845559 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 845560 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49728_solr
   [junit4]   2> 846199 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33000.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 846213 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33000.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 846213 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33000.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 846222 INFO  (jetty-launcher-279-thread-3) [n:127.0.0.1:33000_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node3/.
   [junit4]   2> 846922 INFO  (zkCallback-303-thread-1-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 847269 INFO  (zkCallback-306-thread-1-processing-n:127.0.0.1:56434_solr) [n:127.0.0.1:56434_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 847451 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49728.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 847872 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56434.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848114 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56434.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848114 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56434.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848115 INFO  (jetty-launcher-279-thread-1) [n:127.0.0.1:56434_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node1/.
   [junit4]   2> 848541 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49728.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848562 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 848672 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49728.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 848673 INFO  (jetty-launcher-279-thread-2) [n:127.0.0.1:49728_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node2/.
   [junit4]   2> 849302 INFO  (zkConnectionManagerCallback-312-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 849422 INFO  (zkConnectionManagerCallback-316-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 849423 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 849424 INFO  (SUITE-CollectionTooManyReplicasTest-seed#[EDC45B980F445A0F]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40795/solr ready
   [junit4]   2> 849651 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDownedShards
   [junit4]   2> 849670 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 850145 INFO  (qtp1741440036-1502) [n:127.0.0.1:56434_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params shards=shardstart&replicationFactor=1&collection.configName=conf&maxShardsPerNode=2&name=TooManyReplicasWhenAddingDownedNode&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 850214 INFO  (OverseerThreadFactory-451-thread-1-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.CreateCollectionCmd Create collection TooManyReplicasWhenAddingDownedNode
   [junit4]   2> 850363 INFO  (OverseerStateUpdate-99140736085983240-127.0.0.1:33000_solr-n_0000000000) [n:127.0.0.1:33000_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"TooManyReplicasWhenAddingDownedNode",
   [junit4]   2>   "shard":"shardstart",
   [junit4]   2>   "core":"TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:49728/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 851140 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1&action=CREATE&numShards=1&collection=TooManyReplicasWhenAddingDownedNode&shard=shardstart&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 851140 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 853894 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 854222 INFO  (zkCallback-297-thread-2-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 855275 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
   [junit4]   2> 855373 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.s.IndexSchema [TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] Schema name=minimal
   [junit4]   2> 855375 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 855375 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1' using configuration from collection TooManyReplicasWhenAddingDownedNode, trusted=true
   [junit4]   2> 855376 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49728.solr.core.TooManyReplicasWhenAddingDownedNode.shardstart.replica_n1' (registry 'solr.core.TooManyReplicasWhenAddingDownedNode.shardstart.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5bea840e
   [junit4]   2> 855376 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 855376 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrCore [[TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node2/TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionTooManyReplicasTest_EDC45B980F445A0F-001/tempDir-001/node2/./TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1/data/]
   [junit4]   2> 857419 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 857419 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 857420 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 857420 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 857466 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5dadb795[TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] main]
   [junit4]   2> 857475 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 857475 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 857476 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 857476 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1586251794623758336
   [junit4]   2> 857577 INFO  (searcherExecutor-458-thread-1-processing-n:127.0.0.1:49728_solr x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1 s:shardstart c:TooManyReplicasWhenAddingDownedNode r:core_node2) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrCore [TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] Registered new searcher Searcher@5dadb795[TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:49728/solr/TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1/
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:49728/solr/TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1/ has no replicas
   [junit4]   2> 857582 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 857585 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:49728/solr/TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1/ shardstart
   [junit4]   2> 857944 INFO  (zkCallback-297-thread-2-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 857944 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 857954 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 857970 INFO  (qtp692824956-1484) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1&action=CREATE&numShards=1&collection=TooManyReplicasWhenAddingDownedNode&shard=shardstart&wt=javabin&version=2&replicaType=NRT} status=0 QTime=6830
   [junit4]   2> 858091 INFO  (zkCallback-297-thread-2-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 858091 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 860211 INFO  (qtp1741440036-1502) [n:127.0.0.1:56434_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 860219 INFO  (OverseerCollectionConfigSetProcessor-99140736085983240-127.0.0.1:33000_solr-n_0000000000) [n:127.0.0.1:33000_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 860231 INFO  (qtp1741440036-1502) [n:127.0.0.1:56434_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={shards=shardstart&replicationFactor=1&collection.configName=conf&maxShardsPerNode=2&name=TooManyReplicasWhenAddingDownedNode&router.name=implicit&nrtReplicas=1&action=CREATE&wt=javabin&version=2} status=0 QTime=10085
   [junit4]   2> 860334 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3c531598{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 860338 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1340382632
   [junit4]   2> 860338 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 860339 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@61de3b4e: rootName = solr_49728, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@2ddf3186
   [junit4]   2> 860450 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 860451 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3c26073e: rootName = solr_49728, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2de3c241
   [junit4]   2> 860621 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 860621 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@19dd1edb: rootName = solr_49728, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5ca1b7f2
   [junit4]   2> 860741 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 860750 INFO  (zkCallback-297-thread-2-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/TooManyReplicasWhenAddingDownedNode/state.json] for collection [TooManyReplicasWhenAddingDownedNode] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 860750 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:49728_solr
   [junit4]   2> 860752 INFO  (zkCallback-306-thread-1-processing-n:127.0.0.1:56434_solr) [n:127.0.0.1:56434_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860752 INFO  (zkCallback-315-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860752 INFO  (zkCallback-315-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860761 INFO  (zkCallback-303-thread-2-processing-n:127.0.0.1:33000_solr) [n:127.0.0.1:33000_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860762 INFO  (zkCallback-297-thread-1-processing-n:127.0.0.1:49728_solr) [n:127.0.0.1:49728_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 860763 INFO  (TEST-CollectionTooManyReplicasTest.testDownedShards-seed#[EDC45B980F445A0F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 860986 INFO  (coreCloseExecutor-463-thread-1) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.c.SolrCore [TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@76cd70c2
   [junit4]   2> 860986 INFO  (coreCloseExecutor-463-thread-1) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_shardstart_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.TooManyReplicasWhenAddingDownedNode.shardstart.replica_n1, tag=1993175234
   [junit4]   2> 860986 INFO  (coreCloseExecutor-463-thread-1) [n:127.0.0.1:49728_solr c:TooManyReplicasWhenAddingDownedNode s:shardstart r:core_node2 x:TooManyReplicasWhenAddingDownedNode_sh

[...truncated too long message...]

l, agent id = null] for registry solr.core.localShardsTestColl.shard2.replica_n8 / com.codahale.metrics.MetricRegistry@7ba5f5c0
   [junit4]   2> 465346 INFO  (coreCloseExecutor-357-thread-6) [n:127.0.0.1:45012_solr c:foo s:shard1 r:core_node6 x:foo_shard1_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.foo.shard1.leader, tag=1094082352
   [junit4]   2> 465346 INFO  (coreCloseExecutor-357-thread-5) [n:127.0.0.1:45012_solr c:multicollection1 s:shard1 r:core_node3 x:multicollection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.multicollection1.shard1.leader, tag=53893817
   [junit4]   2> 465346 INFO  (coreCloseExecutor-357-thread-4) [n:127.0.0.1:45012_solr c:localShardsTestColl s:shard3 r:core_node18 x:localShardsTestColl_shard3_replica_n16] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard3.leader, tag=911787225
   [junit4]   2> 465366 INFO  (coreCloseExecutor-357-thread-3) [n:127.0.0.1:45012_solr c:localShardsTestColl s:shard1 r:core_node6 x:localShardsTestColl_shard1_replica_n5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard1.leader, tag=563712393
   [junit4]   2> 465367 INFO  (coreCloseExecutor-355-thread-4) [n:127.0.0.1:53893_solr c:multicollection2 s:shard2 r:core_node4 x:multicollection2_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.multicollection2.shard2.leader, tag=226728099
   [junit4]   2> 465369 INFO  (coreCloseExecutor-355-thread-1) [n:127.0.0.1:53893_solr c:localShardsTestColl s:shard2 r:core_node11 x:localShardsTestColl_shard2_replica_n8] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard2.leader, tag=1241546616
   [junit4]   2> 465369 INFO  (coreCloseExecutor-355-thread-2) [n:127.0.0.1:53893_solr c:localShardsTestColl s:shard3 r:core_node17 x:localShardsTestColl_shard3_replica_n14] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard3.leader, tag=531793602
   [junit4]   2> 465369 INFO  (coreCloseExecutor-355-thread-3) [n:127.0.0.1:53893_solr c:localShardsTestColl s:shard1 r:core_node4 x:localShardsTestColl_shard1_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard1.leader, tag=701305072
   [junit4]   2> 465383 INFO  (coreCloseExecutor-356-thread-8) [n:127.0.0.1:51502_solr c:overwrite s:shard1 r:core_node2 x:overwrite_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.overwrite.shard1.replica_n1, tag=728896613
   [junit4]   2> 465383 INFO  (coreCloseExecutor-356-thread-8) [n:127.0.0.1:51502_solr c:overwrite s:shard1 r:core_node2 x:overwrite_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@727feb3b: rootName = solr_51502, domain = solr.core.overwrite.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.overwrite.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@5e5fa932
   [junit4]   2> 465411 INFO  (coreCloseExecutor-356-thread-8) [n:127.0.0.1:51502_solr c:overwrite s:shard1 r:core_node2 x:overwrite_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.overwrite.shard1.leader, tag=728896613
   [junit4]   2> 465417 INFO  (coreCloseExecutor-356-thread-2) [n:127.0.0.1:51502_solr c:localShardsTestColl s:shard3 r:core_node15 x:localShardsTestColl_shard3_replica_n12] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard3.leader, tag=1759574454
   [junit4]   2> 465418 INFO  (coreCloseExecutor-356-thread-4) [n:127.0.0.1:51502_solr c:localShardsTestColl s:shard1 r:core_node2 x:localShardsTestColl_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard1.leader, tag=90092253
   [junit4]   2> 465418 INFO  (coreCloseExecutor-356-thread-3) [n:127.0.0.1:51502_solr c:localShardsTestColl s:shard2 r:core_node9 x:localShardsTestColl_shard2_replica_n7] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.localShardsTestColl.shard2.leader, tag=927970669
   [junit4]   2> 465418 INFO  (coreCloseExecutor-356-thread-5) [n:127.0.0.1:51502_solr c:multicollection1 s:shard2 r:core_node4 x:multicollection1_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.multicollection1.shard2.leader, tag=1375829709
   [junit4]   2> 465425 INFO  (coreCloseExecutor-356-thread-7) [n:127.0.0.1:51502_solr c:foo s:shard1 r:core_node4 x:foo_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.foo.shard1.leader, tag=1372373832
   [junit4]   2> 465950 INFO  (jetty-closer-149-thread-2) [    ] o.a.s.c.Overseer Overseer (id=99141425872896006-127.0.0.1:53893_solr-n_0000000000) closing
   [junit4]   2> 465951 INFO  (OverseerStateUpdate-99141425872896006-127.0.0.1:53893_solr-n_0000000000) [n:127.0.0.1:53893_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:53893_solr
   [junit4]   2> 466068 INFO  (jetty-closer-149-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@20def09d{/solr,null,UNAVAILABLE}
   [junit4]   2> 466069 WARN  (OverseerAutoScalingTriggerThread-99141425872896006-127.0.0.1:53893_solr-n_0000000000) [n:127.0.0.1:53893_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 466070 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1603899ffae0006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 466329 INFO  (jetty-closer-149-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5fcfe4d0{/solr,null,UNAVAILABLE}
   [junit4]   2> 467849 INFO  (jetty-closer-149-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@48a5fe5{/solr,null,UNAVAILABLE}
   [junit4]   2> 467878 ERROR (SUITE-CloudSolrClientTest-seed#[8FC2650D0D48DB60]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 467878 INFO  (SUITE-CloudSolrClientTest-seed#[8FC2650D0D48DB60]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38623 38623
   [junit4]   2> 467990 INFO  (Thread-115) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38623 38623
   [junit4]   2> 468013 WARN  (Thread-115) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	9	/solr/clusterprops.json
   [junit4]   2> 	9	/solr/aliases.json
   [junit4]   2> 	3	/solr/security.json
   [junit4]   2> 	3	/solr/configs/conf
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	42	/solr/collections/localShardsTestColl/state.json
   [junit4]   2> 	24	/solr/collections/foo/state.json
   [junit4]   2> 	18	/solr/collections/stale_state_test_col/state.json
   [junit4]   2> 	18	/solr/collections/multicollection2/state.json
   [junit4]   2> 	16	/solr/collections/collection1/state.json
   [junit4]   2> 	16	/solr/collections/2nd_collection/state.json
   [junit4]   2> 	14	/solr/collections/nemesis/state.json
   [junit4]   2> 	13	/solr/collections/multicollection1/state.json
   [junit4]   2> 	9	/solr/clusterstate.json
   [junit4]   2> 	6	/solr/collections/overwrite/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	9	/solr/live_nodes
   [junit4]   2> 	9	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-7.x/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.impl.CloudSolrClientTest_8FC2650D0D48DB60-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=Lucene50(blocksize=128), a_t=Lucene50(blocksize=128), title_s=Lucene50(blocksize=128), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), text=BlockTreeOrds(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Asserting), multiDefault=DocValuesFormat(name=Lucene70), title_s=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=344, maxMBSortInHeap=5.990646958084055, sim=RandomSimilarity(queryNorm=false): {}, locale=en-IE, timezone=America/Argentina/San_Juan
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=174697888,total=396886016
   [junit4]   2> NOTE: All tests run in this JVM: [HyperbolicSineEvaluatorTest, JdbcDriverTest, LBHttpSolrClientBuilderTest, SineEvaluatorTest, DocumentAnalysisResponseTest, ExclusiveOrEvaluatorTest, NormalizeEvaluatorTest, UuidEvaluatorTest, LessThanEqualToEvaluatorTest, DivideEvaluatorTest, PowerEvaluatorTest, TestCollectionStateWatchers, SolrExceptionTest, TestSuggesterResponse, CoalesceEvaluatorTest, NoOpResponseParserTest, SolrExampleBinaryTest, ArcCosineEvaluatorTest, TestCloudSolrClientConnections, CloudSolrClientTest]
   [junit4] Completed [73/146 (2!)] on J2 in 162.71s, 16 tests, 1 error <<< FAILURES!

[...truncated 46663 lines...]