You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2018/12/09 02:06:05 UTC

[JENKINS] Lucene-Solr-NightlyTests-7.6 - Build # 22 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.6/22/

No tests ran.

Build Log:
[...truncated 150 lines...]
ERROR: Build step failed with exception
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1036)
		at hudson.FilePath.act(FilePath.java:1025)
		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:233)
		at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
		at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
		at hudson.model.Build$BuildExecution.post2(Build.java:186)
		at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
		at hudson.model.Run.execute(Run.java:1819)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 does not exist.
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:484)
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:461)
	at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:288)
	at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:268)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Build step 'Archive the artifacts' marked build as failure
Recording test results
ERROR: Build step failed with exception
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1036)
		at hudson.FilePath.act(FilePath.java:1025)
		at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)
		at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:136)
		at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:166)
		at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:153)
		at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
		at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
		at hudson.model.Build$BuildExecution.post2(Build.java:186)
		at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
		at hudson.model.Run.execute(Run.java:1819)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 does not exist.
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:484)
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:461)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:141)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Build step 'Publish JUnit test result report' marked build as failure
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-7.6 - Build # 27 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.6/27/

1 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([176FDA4AD2D42496:9F3BE5907C28496E]:0)
	at java.util.Arrays.copyOfRange(Arrays.java:3664)
	at java.lang.String.<init>(String.java:207)
	at java.lang.StringBuilder.toString(StringBuilder.java:407)
	at org.apache.http.client.utils.URLEncodedUtils.urlEncode(URLEncodedUtils.java:547)
	at org.apache.http.client.utils.URLEncodedUtils.encodeFormFields(URLEncodedUtils.java:652)
	at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:404)
	at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:382)
	at org.apache.http.client.entity.UrlEncodedFormEntity.<init>(UrlEncodedFormEntity.java:75)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.fillContentStream(HttpSolrClient.java:513)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:420)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1015)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1031)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:228)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:167)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:669)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:153)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)




Build Log:
[...truncated 14974 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/init-core-data-001
   [junit4]   2> 1937617 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[176FDA4AD2D42496]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1937619 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[176FDA4AD2D42496]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 1937619 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[176FDA4AD2D42496]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1937620 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[176FDA4AD2D42496]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1937624 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1937624 INFO  (Thread-4463) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1937624 INFO  (Thread-4463) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1937626 ERROR (Thread-4463) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1937724 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ZkTestServer start zk server on port:33676
   [junit4]   2> 1937729 INFO  (zkConnectionManagerCallback-4390-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1937735 INFO  (zkConnectionManagerCallback-4392-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1937743 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1937745 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1937747 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1937749 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1937750 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1937752 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1937754 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1937755 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1937757 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1937758 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1937760 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1937762 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 1937940 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 1937942 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1937942 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1937942 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1937942 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@510531d8{/,null,AVAILABLE}
   [junit4]   2> 1937943 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@19ef666e{HTTP/1.1,[http/1.1]}{127.0.0.1:33027}
   [junit4]   2> 1937943 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server Started @1938024ms
   [junit4]   2> 1937943 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/tempDir-001/control/data, hostContext=/, hostPort=33027, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/control-001/cores}
   [junit4]   2> 1937943 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1937943 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1937944 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1937944 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1937944 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1937944 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-16T13:14:43.446Z
   [junit4]   2> 1937946 INFO  (zkConnectionManagerCallback-4394-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1937947 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1937947 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/control-001/solr.xml
   [junit4]   2> 1937952 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1937952 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1937954 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1938334 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33676/solr
   [junit4]   2> 1938336 INFO  (zkConnectionManagerCallback-4398-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1938338 INFO  (zkConnectionManagerCallback-4400-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1938447 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [n:127.0.0.1:33027_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33027_
   [junit4]   2> 1938448 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [n:127.0.0.1:33027_    ] o.a.s.c.Overseer Overseer (id=72457420044828676-127.0.0.1:33027_-n_0000000000) starting
   [junit4]   2> 1938456 INFO  (zkConnectionManagerCallback-4407-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1938459 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [n:127.0.0.1:33027_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33676/solr ready
   [junit4]   2> 1938460 INFO  (OverseerStateUpdate-72457420044828676-127.0.0.1:33027_-n_0000000000) [n:127.0.0.1:33027_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:33027_
   [junit4]   2> 1938460 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [n:127.0.0.1:33027_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33027_
   [junit4]   2> 1938462 INFO  (OverseerStateUpdate-72457420044828676-127.0.0.1:33027_-n_0000000000) [n:127.0.0.1:33027_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1938470 INFO  (zkCallback-4406-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1938496 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [n:127.0.0.1:33027_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1938523 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [n:127.0.0.1:33027_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1938539 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [n:127.0.0.1:33027_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1938539 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [n:127.0.0.1:33027_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1938541 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [n:127.0.0.1:33027_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/control-001/cores
   [junit4]   2> 1938586 INFO  (zkConnectionManagerCallback-4413-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1938588 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1938589 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33676/solr ready
   [junit4]   2> 1938591 INFO  (qtp684498873-17804) [n:127.0.0.1:33027_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33027_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1938600 INFO  (OverseerThreadFactory-6826-thread-1-processing-n:127.0.0.1:33027_) [n:127.0.0.1:33027_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 1938717 INFO  (qtp684498873-17799) [n:127.0.0.1:33027_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1938722 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1938723 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1939748 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1939764 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1939876 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1939896 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 1939896 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1939897 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1939900 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=9, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=69.8916015625, floorSegmentMB=1.7724609375, forceMergeDeletesPctAllowed=22.931552185170048, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=38.42396947966222
   [junit4]   2> 1939902 WARN  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1939976 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1939976 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1939978 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1939978 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1939980 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1077938811, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1939981 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1023191f[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1939983 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1939984 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1939985 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1939986 INFO  (searcherExecutor-6831-thread-1-processing-n:127.0.0.1:33027_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@1023191f[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1939986 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1620014358056665088
   [junit4]   2> 1940003 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 1940007 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1940007 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1940007 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33027/control_collection_shard1_replica_n1/
   [junit4]   2> 1940007 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1940007 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33027/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1940007 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1940012 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33027/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1940015 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1940022 INFO  (qtp684498873-17800) [n:127.0.0.1:33027_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1300
   [junit4]   2> 1940026 INFO  (qtp684498873-17804) [n:127.0.0.1:33027_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1940122 INFO  (zkCallback-4399-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1940603 INFO  (OverseerCollectionConfigSetProcessor-72457420044828676-127.0.0.1:33027_-n_0000000000) [n:127.0.0.1:33027_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1941026 INFO  (qtp684498873-17804) [n:127.0.0.1:33027_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33027_&wt=javabin&version=2} status=0 QTime=2435
   [junit4]   2> 1941032 INFO  (zkConnectionManagerCallback-4418-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1941033 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1941034 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33676/solr ready
   [junit4]   2> 1941034 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1941035 INFO  (qtp684498873-17803) [n:127.0.0.1:33027_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1941038 INFO  (OverseerThreadFactory-6826-thread-2-processing-n:127.0.0.1:33027_) [n:127.0.0.1:33027_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 1941242 WARN  (OverseerThreadFactory-6826-thread-2-processing-n:127.0.0.1:33027_) [n:127.0.0.1:33027_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 1941256 INFO  (qtp684498873-17803) [n:127.0.0.1:33027_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1941256 INFO  (qtp684498873-17803) [n:127.0.0.1:33027_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=220
   [junit4]   2> 1941472 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-1-001 of type NRT
   [junit4]   2> 1941473 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 1941474 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1941474 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1941474 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1941475 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@16f32569{/,null,AVAILABLE}
   [junit4]   2> 1941475 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@432ca69c{HTTP/1.1,[http/1.1]}{127.0.0.1:38074}
   [junit4]   2> 1941475 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server Started @1941556ms
   [junit4]   2> 1941475 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=38074, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-1-001/cores}
   [junit4]   2> 1941475 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1941475 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1941476 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1941476 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1941476 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1941476 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-16T13:14:46.978Z
   [junit4]   2> 1941499 INFO  (zkConnectionManagerCallback-4420-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1941504 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1016ba3bf1d0008, likely client has closed socket
   [junit4]   2> 1941504 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1941504 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-1-001/solr.xml
   [junit4]   2> 1941509 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1941509 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1941510 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1941715 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33676/solr
   [junit4]   2> 1941719 INFO  (zkConnectionManagerCallback-4424-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1941748 INFO  (zkConnectionManagerCallback-4426-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1941758 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1941771 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1941771 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38074_
   [junit4]   2> 1941776 INFO  (zkCallback-4417-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1941776 INFO  (zkCallback-4399-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1941776 INFO  (zkCallback-4406-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1941780 INFO  (zkCallback-4425-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1941806 INFO  (zkConnectionManagerCallback-4433-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1941808 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1941813 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33676/solr ready
   [junit4]   2> 1941813 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1941850 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1941872 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1941872 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1941875 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-1-001/cores
   [junit4]   2> 1941944 INFO  (qtp684498873-17805) [n:127.0.0.1:33027_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:38074_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1941950 INFO  (OverseerCollectionConfigSetProcessor-72457420044828676-127.0.0.1:33027_-n_0000000000) [n:127.0.0.1:33027_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1941959 INFO  (qtp684498873-17803) [n:127.0.0.1:33027_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1941960 INFO  (qtp684498873-17799) [n:127.0.0.1:33027_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 1941962 INFO  (qtp1326092651-17858) [n:127.0.0.1:38074_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1941963 INFO  (OverseerThreadFactory-6826-thread-3-processing-n:127.0.0.1:33027_) [n:127.0.0.1:33027_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:38074_ for creating new replica of shard shard2 for collection collection1
   [junit4]   2> 1941967 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_    x:collection1_shard2_replica_n61] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1942997 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1943016 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema [collection1_shard2_replica_n61] Schema name=test
   [junit4]   2> 1943143 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1943168 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n61' using configuration from collection collection1, trusted=true
   [junit4]   2> 1943169 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n61' (registry 'solr.core.collection1.shard2.replica_n61') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1943169 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [[collection1_shard2_replica_n61] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-1-001/cores/collection1_shard2_replica_n61], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-1-001/cores/collection1_shard2_replica_n61/data/]
   [junit4]   2> 1943173 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=9, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=69.8916015625, floorSegmentMB=1.7724609375, forceMergeDeletesPctAllowed=22.931552185170048, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=38.42396947966222
   [junit4]   2> 1943179 WARN  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1943280 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1943280 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1943282 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1943282 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1943285 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1077938811, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1943286 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.SolrIndexSearcher Opening [Searcher@248b178f[collection1_shard2_replica_n61] main]
   [junit4]   2> 1943288 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1943289 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1943290 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1943290 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1620014361521160192
   [junit4]   2> 1943293 INFO  (searcherExecutor-6845-thread-1-processing-n:127.0.0.1:38074_ x:collection1_shard2_replica_n61 c:collection1 s:shard2) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [collection1_shard2_replica_n61] Registered new searcher Searcher@248b178f[collection1_shard2_replica_n61] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1943301 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node62=0}, version=0}
   [junit4]   2> 1943305 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1943305 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1943305 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38074/collection1_shard2_replica_n61/
   [junit4]   2> 1943305 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1943308 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy http://127.0.0.1:38074/collection1_shard2_replica_n61/ has no replicas
   [junit4]   2> 1943308 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1943321 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38074/collection1_shard2_replica_n61/ shard2
   [junit4]   2> 1943324 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1943329 INFO  (qtp1326092651-17853) [n:127.0.0.1:38074_ c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1362
   [junit4]   2> 1943332 INFO  (qtp684498873-17805) [n:127.0.0.1:33027_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:38074_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1387
   [junit4]   2> 1943558 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-2-001 of type NRT
   [junit4]   2> 1943559 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 1943562 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1943562 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1943562 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1943562 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@503e9c54{/,null,AVAILABLE}
   [junit4]   2> 1943563 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@24d1a788{HTTP/1.1,[http/1.1]}{127.0.0.1:34292}
   [junit4]   2> 1943563 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server Started @1943644ms
   [junit4]   2> 1943563 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=34292, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-2-001/cores}
   [junit4]   2> 1943563 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1943565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1943565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1943565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1943565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1943565 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-16T13:14:49.067Z
   [junit4]   2> 1943567 INFO  (zkConnectionManagerCallback-4436-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1943569 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1943569 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-2-001/solr.xml
   [junit4]   2> 1943574 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1943574 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1943576 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1943848 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33676/solr
   [junit4]   2> 1943850 INFO  (zkConnectionManagerCallback-4440-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1943852 INFO  (zkConnectionManagerCallback-4442-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1943862 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1943868 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1943868 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34292_
   [junit4]   2> 1943870 INFO  (zkCallback-4406-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1943870 INFO  (zkCallback-4441-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1943871 INFO  (zkCallback-4417-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1943871 INFO  (zkCallback-4399-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1943873 INFO  (zkCallback-4425-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1943874 INFO  (zkCallback-4432-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1943899 INFO  (zkConnectionManagerCallback-4449-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1943900 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1943901 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33676/solr ready
   [junit4]   2> 1943902 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1943929 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1943947 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1943947 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1943950 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-2-001/cores
   [junit4]   2> 1943953 INFO  (OverseerCollectionConfigSetProcessor-72457420044828676-127.0.0.1:33027_-n_0000000000) [n:127.0.0.1:33027_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1944009 INFO  (qtp684498873-17799) [n:127.0.0.1:33027_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:34292_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1944015 INFO  (qtp684498873-17805) [n:127.0.0.1:33027_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1944016 INFO  (qtp684498873-17803) [n:127.0.0.1:33027_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1944018 INFO  (qtp1234287692-17896) [n:127.0.0.1:34292_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1944021 INFO  (qtp1326092651-17854) [n:127.0.0.1:38074_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard2.replica_n61:INDEX.sizeInBytes} status=0 QTime=2
   [junit4]   2> 1944023 INFO  (qtp1326092651-17860) [n:127.0.0.1:38074_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1944023 INFO  (OverseerThreadFactory-6826-thread-4-processing-n:127.0.0.1:33027_) [n:127.0.0.1:33027_ c:collection1 s:shard3  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:34292_ for creating new replica of shard shard3 for collection collection1
   [junit4]   2> 1944026 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_    x:collection1_shard3_replica_n63] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1945060 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1945081 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema [collection1_shard3_replica_n63] Schema name=test
   [junit4]   2> 1945204 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1945226 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard3_replica_n63' using configuration from collection collection1, trusted=true
   [junit4]   2> 1945226 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard3.replica_n63' (registry 'solr.core.collection1.shard3.replica_n63') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1945227 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrCore [[collection1_shard3_replica_n63] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-2-001/cores/collection1_shard3_replica_n63], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-2-001/cores/collection1_shard3_replica_n63/data/]
   [junit4]   2> 1945230 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=9, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=69.8916015625, floorSegmentMB=1.7724609375, forceMergeDeletesPctAllowed=22.931552185170048, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=38.42396947966222
   [junit4]   2> 1945232 WARN  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1945300 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1945300 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1945302 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1945302 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1945304 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1077938811, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1945304 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.SolrIndexSearcher Opening [Searcher@4b025f0d[collection1_shard3_replica_n63] main]
   [junit4]   2> 1945305 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1945306 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1945307 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1945308 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1620014363637186560
   [junit4]   2> 1945311 INFO  (searcherExecutor-6859-thread-1-processing-n:127.0.0.1:34292_ x:collection1_shard3_replica_n63 c:collection1 s:shard3) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrCore [collection1_shard3_replica_n63] Registered new searcher Searcher@4b025f0d[collection1_shard3_replica_n63] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1945315 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard3 to Terms{values={core_node64=0}, version=0}
   [junit4]   2> 1945318 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1945318 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1945318 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34292/collection1_shard3_replica_n63/
   [junit4]   2> 1945319 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1945319 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy http://127.0.0.1:34292/collection1_shard3_replica_n63/ has no replicas
   [junit4]   2> 1945319 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1945323 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34292/collection1_shard3_replica_n63/ shard3
   [junit4]   2> 1945381 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1945384 INFO  (qtp1234287692-17891) [n:127.0.0.1:34292_ c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1358
   [junit4]   2> 1945387 INFO  (qtp684498873-17799) [n:127.0.0.1:33027_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:34292_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2} status=0 QTime=1378
   [junit4]   2> 1945578 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-3-001 of type NRT
   [junit4]   2> 1945579 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 1945586 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1945587 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1945587 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1945587 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@76caa47a{/,null,AVAILABLE}
   [junit4]   2> 1945588 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d387304{HTTP/1.1,[http/1.1]}{127.0.0.1:44971}
   [junit4]   2> 1945588 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server Started @1945669ms
   [junit4]   2> 1945588 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=44971, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-3-001/cores}
   [junit4]   2> 1945588 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1945593 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1945593 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 1945593 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1945593 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1945594 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-16T13:14:51.096Z
   [junit4]   2> 1945604 INFO  (zkConnectionManagerCallback-4452-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1945606 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1945606 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-3-001/solr.xml
   [junit4]   2> 1945619 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1945619 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1945621 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1945833 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33676/solr
   [junit4]   2> 1945835 INFO  (zkConnectionManagerCallback-4456-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1945838 INFO  (zkConnectionManagerCallback-4458-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1945846 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1945852 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1945852 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44971_
   [junit4]   2> 1945854 INFO  (zkCallback-4406-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1945854 INFO  (zkCallback-4441-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1945854 INFO  (zkCallback-4417-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1945854 INFO  (zkCallback-4448-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1945854 INFO  (zkCallback-4425-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1945855 INFO  (zkCallback-4432-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1945855 INFO  (zkCallback-4399-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1945864 INFO  (zkCallback-4457-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1945885 INFO  (zkConnectionManagerCallback-4465-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1945886 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1945887 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33676/solr ready
   [junit4]   2> 1945888 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 1945919 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1945938 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1945938 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1945941 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-3-001/cores
   [junit4]   2> 1946037 INFO  (OverseerCollectionConfigSetProcessor-72457420044828676-127.0.0.1:33027_-n_0000000000) [n:127.0.0.1:33027_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1946039 INFO  (qtp684498873-17805) [n:127.0.0.1:33027_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:44971_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1946044 INFO  (qtp684498873-17799) [n:127.0.0.1:33027_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1946046 INFO  (qtp684498873-17803) [n:127.0.0.1:33027_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1946046 INFO  (qtp1234287692-17892) [n:127.0.0.1:34292_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard3.replica_n63:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1946050 INFO  (qtp1234287692-17892) [n:127.0.0.1:34292_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1946051 INFO  (qtp1326092651-17856) [n:127.0.0.1:38074_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.collection1.shard2.replica_n61:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 1946053 INFO  (qtp1326092651-17858) [n:127.0.0.1:38074_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1946055 INFO  (qtp764383167-17933) [n:127.0.0.1:44971_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 1946056 INFO  (OverseerThreadFactory-6826-thread-5-processing-n:127.0.0.1:33027_) [n:127.0.0.1:33027_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:44971_ for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 1946059 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_    x:collection1_shard1_replica_n65] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n65&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1947082 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 1947098 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.IndexSchema [collection1_shard1_replica_n65] Schema name=test
   [junit4]   2> 1947211 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1947229 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n65' using configuration from collection collection1, trusted=true
   [junit4]   2> 1947229 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n65' (registry 'solr.core.collection1.shard1.replica_n65') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47567a5d
   [junit4]   2> 1947230 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrCore [[collection1_shard1_replica_n65] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-3-001/cores/collection1_shard1_replica_n65], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-3-001/cores/collection1_shard1_replica_n65/data/]
   [junit4]   2> 1947233 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=9, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=69.8916015625, floorSegmentMB=1.7724609375, forceMergeDeletesPctAllowed=22.931552185170048, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=38.42396947966222
   [junit4]   2> 1947235 WARN  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1947302 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1947302 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1947304 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1947304 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1947306 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1077938811, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1947306 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.SolrIndexSearcher Opening [Searcher@3a865027[collection1_shard1_replica_n65] main]
   [junit4]   2> 1947308 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1947308 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1947309 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1947310 INFO  (searcherExecutor-6873-thread-1-processing-n:127.0.0.1:44971_ x:collection1_shard1_replica_n65 c:collection1 s:shard1) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrCore [collection1_shard1_replica_n65] Registered new searcher Searcher@3a865027[collection1_shard1_replica_n65] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1947310 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1620014365736435712
   [junit4]   2> 1947316 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node66=0}, version=0}
   [junit4]   2> 1947319 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1947319 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1947319 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44971/collection1_shard1_replica_n65/
   [junit4]   2> 1947319 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1947319 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy http://127.0.0.1:44971/collection1_shard1_replica_n65/ has no replicas
   [junit4]   2> 1947319 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1947324 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44971/collection1_shard1_replica_n65/ shard1
   [junit4]   2> 1947328 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1947331 INFO  (qtp764383167-17928) [n:127.0.0.1:44971_ c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n65&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1271
   [junit4]   2> 1947333 INFO  (qtp684498873-17805) [n:127.0.0.1:33027_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:44971_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1293
   [junit4]   2> 1947511 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_176FDA4AD2D42496-001/shard-4-001 of type NRT
   [junit4]   2> 1947512 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 1947514 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1947514 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1947514 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1947515 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@26ca6b3a{/,null,AVAILABLE}
   [junit4]   2> 1947515 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1d69f5ff{HTTP/1.1,[http/1.1]}{127.0.0.1:33521}
   [junit4]   2> 1947515 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.e.j.s.Server Started @1947596ms
   [junit4]   2> 1947515 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[176FDA4AD2D42496]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyT

[...truncated too long message...]

128), 29parent_f1_s=PostingsFormat(name=Memory), 33parent_f1_s=PostingsFormat(name=Memory), 39parent_f1_s=FSTOrd50, 3child0_s=Lucene50(blocksize=128), 4child3_s=Lucene50(blocksize=128), 12parent_f1_s=Lucene50(blocksize=128), 22parent_f1_s=PostingsFormat(name=Memory), 1grand0_s=PostingsFormat(name=Memory), 45parent_f2_s=PostingsFormat(name=Memory), 7parent_f1_s=PostingsFormat(name=Memory), 2child1_s=Lucene50(blocksize=128), 32parent_f1_s=PostingsFormat(name=Memory), 3parent_f2_s=PostingsFormat(name=Memory), 42parent_f1_s=FSTOrd50, 6grand3_s=PostingsFormat(name=Memory), 31parent_f2_s=Lucene50(blocksize=128), 41parent_f2_s=PostingsFormat(name=Memory), 5grand2_s=Lucene50(blocksize=128), 11parent_f2_s=PostingsFormat(name=Memory), 21parent_f2_s=FSTOrd50, 4grand1_s=PostingsFormat(name=Memory), 51parent_f2_s=PostingsFormat(name=Memory), 7grand4_s=Lucene50(blocksize=128), 3grand0_s=Lucene50(blocksize=128), 40parent_f1_s=PostingsFormat(name=Memory), 50parent_f1_s=PostingsFormat(name=Memory), 9parent_f1_s=Lucene50(blocksize=128), 30parent_f1_s=Lucene50(blocksize=128), 14parent_f2_s=FSTOrd50, 20parent_f1_s=FSTOrd50, 1parent_f2_s=FSTOrd50, 7grand0_s=Lucene50(blocksize=128), text_t=PostingsFormat(name=Memory), 24parent_f2_s=Lucene50(blocksize=128), 44parent_f2_s=PostingsFormat(name=Memory), 34parent_f2_s=PostingsFormat(name=Memory), 13parent_f2_s=Lucene50(blocksize=128), type_s=PostingsFormat(name=Memory), 23parent_f2_s=PostingsFormat(name=Memory), 43parent_f2_s=FSTOrd50, 10parent_f1_s=PostingsFormat(name=Memory), 33parent_f2_s=PostingsFormat(name=Memory), 22parent_f2_s=PostingsFormat(name=Memory), 4child0_s=FSTOrd50, 12parent_f2_s=PostingsFormat(name=Memory), 42parent_f2_s=Lucene50(blocksize=128), 32parent_f2_s=FSTOrd50, 31parent_f1_s=FSTOrd50, 5grand1_s=PostingsFormat(name=Memory), 6grand4_s=FSTOrd50, 11parent_f1_s=PostingsFormat(name=Memory), 51parent_f1_s=PostingsFormat(name=Memory), 21parent_f1_s=PostingsFormat(name=Memory), 4grand2_s=FSTOrd50, 7grand3_s=PostingsFormat(name=Memory), 41parent_f1_s=Lucene50(blocksize=128), 37parent_f1_s=PostingsFormat(name=Memory), 40parent_f2_s=PostingsFormat(name=Memory), 50parent_f2_s=FSTOrd50, 47parent_f1_s=PostingsFormat(name=Memory), 14parent_f1_s=PostingsFormat(name=Memory), 20parent_f2_s=Lucene50(blocksize=128), 34parent_f1_s=Lucene50(blocksize=128), 24parent_f1_s=FSTOrd50, 44parent_f1_s=PostingsFormat(name=Memory), 30parent_f2_s=PostingsFormat(name=Memory), 5parent_f1_s=Lucene50(blocksize=128), 6grand0_s=FSTOrd50, 46parent_f2_s=Lucene50(blocksize=128), 10parent_f2_s=FSTOrd50, 8parent_f1_s=FSTOrd50, 27parent_f2_s=PostingsFormat(name=Memory), 2parent_f2_s=Lucene50(blocksize=128), 4child1_s=PostingsFormat(name=Memory), 2grand1_s=Lucene50(blocksize=128), 17parent_f2_s=Lucene50(blocksize=128), 3child2_s=PostingsFormat(name=Memory), 16parent_f2_s=FSTOrd50, 26parent_f2_s=PostingsFormat(name=Memory), 1child0_s=PostingsFormat(name=Memory), 36parent_f2_s=FSTOrd50, 7grand6_s=PostingsFormat(name=Memory), 18parent_f1_s=PostingsFormat(name=Memory), 28parent_f1_s=PostingsFormat(name=Memory), 5grand0_s=PostingsFormat(name=Memory), 6grand5_s=Lucene50(blocksize=128), 7grand2_s=PostingsFormat(name=Memory), 48parent_f1_s=PostingsFormat(name=Memory), 15parent_f1_s=PostingsFormat(name=Memory), 4grand3_s=Lucene50(blocksize=128), 6parent_f1_s=PostingsFormat(name=Memory), 25parent_f1_s=PostingsFormat(name=Memory), 0parent_f1_s=FSTOrd50, 3grand2_s=PostingsFormat(name=Memory), 35parent_f1_s=FSTOrd50, 19parent_f2_s=PostingsFormat(name=Memory), 4parent_f2_s=PostingsFormat(name=Memory), 37parent_f2_s=PostingsFormat(name=Memory), 47parent_f2_s=FSTOrd50, 38parent_f1_s=Lucene50(blocksize=128), 5grand4_s=PostingsFormat(name=Memory), 6grand1_s=Lucene50(blocksize=128), 5parent_f2_s=FSTOrd50, 46parent_f1_s=FSTOrd50, 16parent_f1_s=Lucene50(blocksize=128), 8parent_f2_s=PostingsFormat(name=Memory), 39parent_f2_s=Lucene50(blocksize=128), 29parent_f2_s=PostingsFormat(name=Memory), 2parent_f1_s=PostingsFormat(name=Memory), 4child2_s=PostingsFormat(name=Memory), rnd_b=Lucene50(blocksize=128), 27parent_f1_s=Lucene50(blocksize=128), 45parent_f1_s=Lucene50(blocksize=128), 17parent_f1_s=PostingsFormat(name=Memory), 2grand0_s=PostingsFormat(name=Memory), 49parent_f2_s=PostingsFormat(name=Memory), 3child1_s=FSTOrd50, 7parent_f2_s=PostingsFormat(name=Memory), 26parent_f1_s=PostingsFormat(name=Memory), 2child0_s=FSTOrd50, 3parent_f1_s=PostingsFormat(name=Memory), 36parent_f1_s=PostingsFormat(name=Memory)}, docValues:{_version_=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Lucene70), text_t=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1190, maxMBSortInHeap=6.437865357597358, sim=RandomSimilarity(queryNorm=false): {}, locale=el, timezone=Asia/Kabul
   [junit4]   2> NOTE: Linux 4.4.0-137-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=127359600,total=512229376
   [junit4]   2> NOTE: All tests run in this JVM: [DataDrivenBlockJoinTest, TestCollapseQParserPlugin, TestMissingGroups, ZkStateWriterTest, TestLRUCache, TestSystemIdResolver, TestNumericTerms32, ShufflingReplicaListTransformerTest, TestCloudSearcherWarming, DocumentAnalysisRequestHandlerTest, NotRequiredUniqueKeyTest, DistributedFacetPivotWhiteBoxTest, UpdateLogTest, TestHighlightDedupGrouping, TestFieldTypeResource, DocValuesMissingTest, TestPHPSerializedResponseWriter, TestQuerySenderListener, TestDownShardTolerantSearch, TestInPlaceUpdatesDistrib, TestCloudManagedSchema, DocValuesNotIndexedTest, TestCollationFieldDocValues, HdfsBasicDistributedZkTest, TestRawTransformer, DocExpirationUpdateProcessorFactoryTest, DisMaxRequestHandlerTest, DistributedQueryComponentOptimizationTest, ReplaceNodeNoTargetTest, VMParamsZkACLAndCredentialsProvidersTest, SynonymTokenizerTest, TestMacros, TestFunctionQuery, SolrMetricReporterTest, TestFastOutputStream, TestCloudPivotFacet, TestXmlQParser, TestConfigSets, TestDynamicLoading, TestScoreJoinQPNoScore, PropertiesRequestHandlerTest, TestBinaryResponseWriter, BufferStoreTest, OverseerTaskQueueTest, BooleanFieldTest, HighlighterTest, MBeansHandlerTest, BlockJoinFacetDistribTest, TestPointFields, CacheHeaderTest, TestFieldCacheReopen, NoCacheHeaderTest, TestChildDocTransformer, TestExpandComponent, ActionThrottleTest, TimeZoneUtilsTest, TestNonDefinedSimilarityFactory, ConcurrentDeleteAndCreateCollectionTest, TestIndexSearcher, IndexSizeTriggerTest, BasicFunctionalityTest, TestJsonFacetsStatsParsing, NodeLostTriggerTest, TestMultiValuedNumericRangeQuery, TestPseudoReturnFields, SuggestComponentContextFilterQueryTest, LIRRollingUpdatesTest, SpellPossibilityIteratorTest, TestPrepRecovery, TestSimComputePlanAction, XmlUpdateRequestHandlerTest, TestSolr4Spatial2, DistributedFacetPivotLongTailTest, RestartWhileUpdatingTest, PluginInfoTest, AutoAddReplicasPlanActionTest, HdfsRestartWhileUpdatingTest, RequestHandlersTest, BasicDistributedZkTest, TestUnInvertedFieldException, MoreLikeThisHandlerTest, ExitableDirectoryReaderTest, TestSolrDeletionPolicy1, DistribJoinFromCollectionTest, TestSubQueryTransformerCrossCore, CheckHdfsIndexTest, TestUtilizeNode, TestImpersonationWithHadoopAuth, TestSweetSpotSimilarityFactory, TestRecovery, TestOverriddenPrefixQueryForCustomFieldType, TestGeoJSONResponseWriter, TaggerTest, TestStressCloudBlindAtomicUpdates, TestRuleBasedAuthorizationPlugin, TestLegacyFieldReuse, TestManagedSchemaAPI, TestIBSimilarityFactory, TestDocTermOrds, MaxSizeAutoCommitTest, TestSlowCompositeReaderWrapper, IndexSchemaTest, TestReloadDeadlock, MetricsConfigTest, PrimitiveFieldTypeTest, TestPhraseSuggestions, TestTrackingShardHandlerFactory, TestPolicyCloud, SolrInfoBeanTest, SaslZkACLProviderTest, TestCoreAdminApis, TestNamedUpdateProcessors, FileUtilsTest, TestCSVLoader, MultiSolrCloudTestCaseTest, CurrencyFieldTypeTest, HLLSerializationTest, WordBreakSolrSpellCheckerTest, TestReqParamsAPI, TestTlogReplica, TestCursorMarkWithoutUniqueKey, TermVectorComponentTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestRTimerTree, MinimalSchemaTest, SolrCloudExampleTest, TestStressReorder, TestManagedResourceStorage, TestEmbeddedSolrServerSchemaAPI, RankQueryTest, TestTolerantSearch, TestInfoStreamLogging, TestReversedWildcardFilterFactory, NodeAddedTriggerIntegrationTest, RegexBoostProcessorTest, OutputWriterTest, TestDistributedGrouping, TestRandomFaceting, TestSimpleTrackingShardHandler, TestCharFilters, TokenizerChainTest, TestEmbeddedSolrServerAdminHandler, TestJettySolrRunner, AliasIntegrationTest, ChaosMonkeySafeLeaderWithPullReplicasTest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DeleteNodeTest, DeleteShardTest, DeleteStatusTest, DistributedQueueTest, ForceLeaderTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed [506/836 (1!)] on J1 in 762.48s, 1 test, 1 error <<< FAILURES!

[...truncated 1154 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/temp/junit4-J1-20181216_124225_3483766793904151649004.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/heapdumps/java_pid10312.hprof ...
   [junit4] Heap dump file created [569788600 bytes in 3.816 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 9008 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/build.xml:651: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid10312.hprof

Total time: 256 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-7.6 - Build # 26 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.6/26/

1 tests failed.
FAILED:  org.apache.lucene.document.TestLatLonLineShapeQueries.testRandomBig

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([37675F0D7C15A9FC:B0302282ED4CD57C]:0)
	at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
	at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
	at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
	at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
	at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
	at org.apache.lucene.util.bkd.OfflinePointReader.split(OfflinePointReader.java:221)
	at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1846)
	at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:1025)
	at org.apache.lucene.index.RandomCodec$1$1.writeField(RandomCodec.java:142)
	at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:62)
	at org.apache.lucene.codecs.PointsWriter.merge(PointsWriter.java:191)
	at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:145)
	at org.apache.lucene.codecs.asserting.AssertingPointsFormat$AssertingPointsWriter.merge(AssertingPointsFormat.java:142)
	at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:188)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2170)
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5103)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1612)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1228)
	at org.apache.lucene.document.BaseLatLonShapeTestCase.indexRandomShapes(BaseLatLonShapeTestCase.java:255)
	at org.apache.lucene.document.BaseLatLonShapeTestCase.verify(BaseLatLonShapeTestCase.java:231)
	at org.apache.lucene.document.BaseLatLonShapeTestCase.doTestRandom(BaseLatLonShapeTestCase.java:212)
	at org.apache.lucene.document.BaseLatLonShapeTestCase.testRandomBig(BaseLatLonShapeTestCase.java:188)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)




Build Log:
[...truncated 10510 lines...]
   [junit4] Suite: org.apache.lucene.document.TestLatLonLineShapeQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLatLonLineShapeQueries -Dtests.method=testRandomBig -Dtests.seed=37675F0D7C15A9FC -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data/enwiki.random.lines.txt -Dtests.locale=uk -Dtests.timezone=Pacific/Fakaofo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    342s J0 | TestLatLonLineShapeQueries.testRandomBig <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([37675F0D7C15A9FC:B0302282ED4CD57C]:0)
   [junit4]    > 	at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:84)
   [junit4]    > 	at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:57)
   [junit4]    > 	at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:168)
   [junit4]    > 	at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:154)
   [junit4]    > 	at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:141)
   [junit4]    > 	at org.apache.lucene.util.bkd.OfflinePointReader.split(OfflinePointReader.java:221)
   [junit4]    > 	at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1846)
   [junit4]    > 	at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:1025)
   [junit4]    > 	at org.apache.lucene.index.RandomCodec$1$1.writeField(RandomCodec.java:142)
   [junit4]    > 	at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:62)
   [junit4]    > 	at org.apache.lucene.codecs.PointsWriter.merge(PointsWriter.java:191)
   [junit4]    > 	at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:145)
   [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingPointsFormat$AssertingPointsWriter.merge(AssertingPointsFormat.java:142)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:188)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4446)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4068)
   [junit4]    > 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2170)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5103)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1612)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1228)
   [junit4]    > 	at org.apache.lucene.document.BaseLatLonShapeTestCase.indexRandomShapes(BaseLatLonShapeTestCase.java:255)
   [junit4]    > 	at org.apache.lucene.document.BaseLatLonShapeTestCase.verify(BaseLatLonShapeTestCase.java:231)
   [junit4]    > 	at org.apache.lucene.document.BaseLatLonShapeTestCase.doTestRandom(BaseLatLonShapeTestCase.java:212)
   [junit4]    > 	at org.apache.lucene.document.BaseLatLonShapeTestCase.testRandomBig(BaseLatLonShapeTestCase.java:188)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/build/sandbox/test/J0/temp/lucene.document.TestLatLonLineShapeQueries_37675F0D7C15A9FC-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=Memory)}, docValues:{shape=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=888, maxMBSortInHeap=6.717108673289789, sim=RandomSimilarity(queryNorm=true): {}, locale=uk, timezone=Pacific/Fakaofo
   [junit4]   2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=1,free=103976648,total=489684992
   [junit4]   2> NOTE: All tests run in this JVM: [TestIDVersionPostingsFormat, FuzzyLikeThisQueryTest, TestIntervals, TestNearest, TestFloatPointNearestNeighbor, TestHalfFloatPoint, TestLatLonLineShapeQueries]
   [junit4] Completed [13/22 (1!)] on J0 in 1113.85s, 4 tests, 1 error <<< FAILURES!

[...truncated 23 lines...]
   [junit4] JVM J0: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/build/sandbox/test/temp/junit4-J0-20181215_074222_5985816259583619156449.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/heapdumps/java_pid11464.hprof ...
   [junit4] Heap dump file created [455619805 bytes in 2.734 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 14924 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/build.xml:651: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid11464.hprof

Total time: 489 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-7.6 - Build # 25 - Failure

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.6/25/

No tests ran.

Build Log:
[...truncated 6 lines...]
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
	at hudson.scm.SCM.checkout(SCM.java:504)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1794)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout: 
stderr: error: failed to write new configuration file .git/config.lock

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1970)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1966)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor550.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
		at com.sun.proxy.$Proxy117.setRemoteUrl(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:876)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
		at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1794)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
	at hudson.scm.SCM.checkout(SCM.java:504)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1794)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout: 
stderr: error: failed to write new configuration file .git/config.lock

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1970)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1966)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor550.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
		at com.sun.proxy.$Proxy117.setRemoteUrl(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:876)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
		at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1794)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git://git.apache.org/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
	at hudson.scm.SCM.checkout(SCM.java:504)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1794)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url git://git.apache.org/lucene-solr.git" returned status code 4:
stdout: 
stderr: error: failed to write new configuration file .git/config.lock

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1970)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1966)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1609)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1243)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor550.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
		at com.sun.proxy.$Proxy117.setRemoteUrl(Unknown Source)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:876)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
		at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1794)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Archiving artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/build/analysis/common/test/TEST-org.apache.lucene.analysis.ar.TestArabicAnalyzer.xml is 5 days 5 hr old

Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-7.6 - Build # 24 - Still unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.6/24/

5 tests failed.
FAILED:  org.apache.solr.cloud.RestartWhileUpdatingTest.test

Error Message:
There are still nodes recoverying - waited for 320 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 320 seconds
	at __randomizedtesting.SeedInfo.seed([21858C97DB913823:A9D1B34D756D55DB]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:185)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:920)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1477)
	at org.apache.solr.cloud.RestartWhileUpdatingTest.test(RestartWhileUpdatingTest.java:145)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RestartWhileUpdatingTest

Error Message:
7 threads leaked from SUITE scope at org.apache.solr.cloud.RestartWhileUpdatingTest:     1) Thread[id=133282, name=searcherExecutor-12633-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=133050, name=searcherExecutor-12557-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=133007, name=searcherExecutor-12543-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=133209, name=searcherExecutor-12605-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=133087, name=searcherExecutor-12571-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=133130, name=searcherExecutor-12585-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=133247, name=searcherExecutor-12619-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 7 threads leaked from SUITE scope at org.apache.solr.cloud.RestartWhileUpdatingTest: 
   1) Thread[id=133282, name=searcherExecutor-12633-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=133050, name=searcherExecutor-12557-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=133007, name=searcherExecutor-12543-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=133209, name=searcherExecutor-12605-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=133087, name=searcherExecutor-12571-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=133130, name=searcherExecutor-12585-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=133247, name=searcherExecutor-12619-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([21858C97DB913823]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RestartWhileUpdatingTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=133282, name=searcherExecutor-12633-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=133050, name=searcherExecutor-12557-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=133007, name=searcherExecutor-12543-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=133209, name=searcherExecutor-12605-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    5) Thread[id=133087, name=searcherExecutor-12571-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    6) Thread[id=133130, name=searcherExecutor-12585-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    7) Thread[id=133247, name=searcherExecutor-12619-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=133282, name=searcherExecutor-12633-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=133050, name=searcherExecutor-12557-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=133007, name=searcherExecutor-12543-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=133209, name=searcherExecutor-12605-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   5) Thread[id=133087, name=searcherExecutor-12571-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   6) Thread[id=133130, name=searcherExecutor-12585-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   7) Thread[id=133247, name=searcherExecutor-12619-thread-1, state=WAITING, group=TGRP-RestartWhileUpdatingTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([21858C97DB913823]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamDecoratorTest

Error Message:
ObjectTracker found 2 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)  at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)  at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:957)  at org.apache.solr.client.solrj.io.SolrClientCache.getHttpSolrClient(SolrClientCache.java:83)  at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:105)  at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:505)  at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:494)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)  at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)  at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)  at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:957)  at org.apache.solr.client.solrj.io.SolrClientCache.getHttpSolrClient(SolrClientCache.java:83)  at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:105)  at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:505)  at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:494)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)
	at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:957)
	at org.apache.solr.client.solrj.io.SolrClientCache.getHttpSolrClient(SolrClientCache.java:83)
	at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:105)
	at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:505)
	at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:494)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)
	at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)
	at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:957)
	at org.apache.solr.client.solrj.io.SolrClientCache.getHttpSolrClient(SolrClientCache.java:83)
	at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:105)
	at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:505)
	at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:494)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


	at __randomizedtesting.SeedInfo.seed([40C63D225D936598]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:304)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamDecoratorTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest:     1) Thread[id=1146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=1147, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest: 
   1) Thread[id=1146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=1147, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([40C63D225D936598]:0)




Build Log:
[...truncated 15138 lines...]
   [junit4] Suite: org.apache.solr.cloud.RestartWhileUpdatingTest
   [junit4]   2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/init-core-data-001
   [junit4]   2> 3563173 WARN  (SUITE-RestartWhileUpdatingTest-seed#[21858C97DB913823]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=18 numCloses=18
   [junit4]   2> 3563173 INFO  (SUITE-RestartWhileUpdatingTest-seed#[21858C97DB913823]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 3563187 INFO  (SUITE-RestartWhileUpdatingTest-seed#[21858C97DB913823]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 3563188 INFO  (SUITE-RestartWhileUpdatingTest-seed#[21858C97DB913823]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3563188 INFO  (SUITE-RestartWhileUpdatingTest-seed#[21858C97DB913823]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 3563202 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3563207 INFO  (Thread-5282) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3563207 INFO  (Thread-5282) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3563223 ERROR (Thread-5282) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3563307 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.ZkTestServer start zk server on port:35241
   [junit4]   2> 3563309 INFO  (zkConnectionManagerCallback-67719-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3563312 INFO  (zkConnectionManagerCallback-67721-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3563324 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3563326 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 3563327 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3563336 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3563337 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3563338 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3563348 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3563349 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3563350 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3563351 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3563352 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3563362 INFO  (zkConnectionManagerCallback-67725-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3563363 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 3563804 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 3563805 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3563805 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3563805 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3563822 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@687290ac{/,null,AVAILABLE}
   [junit4]   2> 3563862 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@60c279ba{SSL,[ssl, http/1.1]}{127.0.0.1:44855}
   [junit4]   2> 3563862 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.Server Started @3564118ms
   [junit4]   2> 3563863 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=44855, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/control-001/cores}
   [junit4]   2> 3563863 ERROR (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3563863 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3563863 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 3563863 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3563863 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3563863 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-12T19:24:04.459Z
   [junit4]   2> 3563976 INFO  (zkConnectionManagerCallback-67727-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3563977 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3563977 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/control-001/solr.xml
   [junit4]   2> 3563998 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3563998 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3564000 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3564246 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35241/solr
   [junit4]   2> 3564247 INFO  (zkConnectionManagerCallback-67731-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3564261 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1064822cfd30004, likely client has closed socket
   [junit4]   2> 3564261 INFO  (zkConnectionManagerCallback-67733-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3564476 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [n:127.0.0.1:44855_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44855_
   [junit4]   2> 3564477 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [n:127.0.0.1:44855_    ] o.a.s.c.Overseer Overseer (id=73825758250991621-127.0.0.1:44855_-n_0000000000) starting
   [junit4]   2> 3564499 INFO  (zkConnectionManagerCallback-67740-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3564500 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [n:127.0.0.1:44855_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35241/solr ready
   [junit4]   2> 3564501 INFO  (OverseerStateUpdate-73825758250991621-127.0.0.1:44855_-n_0000000000) [n:127.0.0.1:44855_    ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:44855_
   [junit4]   2> 3564502 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [n:127.0.0.1:44855_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44855_
   [junit4]   2> 3564576 INFO  (zkCallback-67739-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3564576 INFO  (OverseerStateUpdate-73825758250991621-127.0.0.1:44855_-n_0000000000) [n:127.0.0.1:44855_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3564635 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [n:127.0.0.1:44855_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3564752 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [n:127.0.0.1:44855_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39
   [junit4]   2> 3564789 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [n:127.0.0.1:44855_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39
   [junit4]   2> 3564789 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [n:127.0.0.1:44855_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39
   [junit4]   2> 3564803 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [n:127.0.0.1:44855_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/control-001/cores
   [junit4]   2> 3564918 INFO  (zkConnectionManagerCallback-67746-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3564919 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3564938 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35241/solr ready
   [junit4]   2> 3565073 INFO  (qtp800082899-132966) [n:127.0.0.1:44855_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44855_&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3565167 INFO  (OverseerThreadFactory-12538-thread-1-processing-n:127.0.0.1:44855_) [n:127.0.0.1:44855_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 3565276 INFO  (qtp800082899-132969) [n:127.0.0.1:44855_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3565289 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_    x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3565289 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_    x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3566313 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3566344 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 3566552 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 3566628 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
   [junit4]   2> 3566629 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39
   [junit4]   2> 3566629 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 3566632 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@34d4df81
   [junit4]   2> 3566652 WARN  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3566796 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3566812 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3566813 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 30000ms; 
   [junit4]   2> 3566813 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 
   [junit4]   2> 3566815 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3566816 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@16cbc098[control_collection_shard1_replica_n1] main]
   [junit4]   2> 3566817 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3566817 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3566818 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3566818 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1619675208654782464
   [junit4]   2> 3566819 INFO  (searcherExecutor-12543-thread-1-processing-n:127.0.0.1:44855_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@16cbc098[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3566835 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 3566838 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3566838 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3566838 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:44855/control_collection_shard1_replica_n1/
   [junit4]   2> 3566838 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3566839 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:44855/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 3566839 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3566842 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:44855/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 3566944 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3566946 INFO  (qtp800082899-132968) [n:127.0.0.1:44855_ c:control_collection s:shard1  x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1657
   [junit4]   2> 3566948 INFO  (qtp800082899-132966) [n:127.0.0.1:44855_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3567046 INFO  (zkCallback-67732-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3567169 INFO  (OverseerCollectionConfigSetProcessor-73825758250991621-127.0.0.1:44855_-n_0000000000) [n:127.0.0.1:44855_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3567949 INFO  (qtp800082899-132966) [n:127.0.0.1:44855_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44855_&wt=javabin&version=2} status=0 QTime=2875
   [junit4]   2> 3567953 INFO  (zkConnectionManagerCallback-67751-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3567953 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3567954 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35241/solr ready
   [junit4]   2> 3567954 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3567976 INFO  (qtp800082899-132969) [n:127.0.0.1:44855_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3567979 INFO  (OverseerThreadFactory-12538-thread-2-processing-n:127.0.0.1:44855_) [n:127.0.0.1:44855_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 3568185 WARN  (OverseerThreadFactory-12538-thread-2-processing-n:127.0.0.1:44855_) [n:127.0.0.1:44855_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 3568187 INFO  (qtp800082899-132969) [n:127.0.0.1:44855_    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3568187 INFO  (qtp800082899-132969) [n:127.0.0.1:44855_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=210
   [junit4]   2> 3568486 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/shard-1-001 of type NRT
   [junit4]   2> 3568488 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 3568492 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3568493 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3568493 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3568493 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7a94af73{/,null,AVAILABLE}
   [junit4]   2> 3568493 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@243fabf8{SSL,[ssl, http/1.1]}{127.0.0.1:41368}
   [junit4]   2> 3568493 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.Server Started @3568749ms
   [junit4]   2> 3568493 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=41368, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/shard-1-001/cores}
   [junit4]   2> 3568494 ERROR (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3568494 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3568494 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 3568494 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3568494 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3568494 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-12T19:24:09.090Z
   [junit4]   2> 3568507 INFO  (zkConnectionManagerCallback-67753-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3568508 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3568508 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/shard-1-001/solr.xml
   [junit4]   2> 3568523 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3568523 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3568524 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3568902 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35241/solr
   [junit4]   2> 3568904 INFO  (zkConnectionManagerCallback-67757-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3568905 INFO  (zkConnectionManagerCallback-67759-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3568911 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3568923 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3568923 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41368_
   [junit4]   2> 3568948 INFO  (zkCallback-67739-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3568948 INFO  (zkCallback-67732-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3568948 INFO  (zkCallback-67750-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3568949 INFO  (zkCallback-67758-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3568990 INFO  (zkConnectionManagerCallback-67766-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3568991 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3569000 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35241/solr ready
   [junit4]   2> 3569000 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3569049 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39
   [junit4]   2> 3569071 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39
   [junit4]   2> 3569071 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39
   [junit4]   2> 3569072 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/shard-1-001/cores
   [junit4]   2> 3569167 INFO  (qtp1772669569-133022) [n:127.0.0.1:41368_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41368_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3569169 INFO  (OverseerCollectionConfigSetProcessor-73825758250991621-127.0.0.1:44855_-n_0000000000) [n:127.0.0.1:44855_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3569174 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3569177 INFO  (qtp800082899-132967) [n:127.0.0.1:44855_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.control_collection.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
   [junit4]   2> 3569178 INFO  (qtp800082899-132969) [n:127.0.0.1:44855_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3569178 INFO  (OverseerThreadFactory-12538-thread-3-processing-n:127.0.0.1:44855_) [n:127.0.0.1:44855_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:41368_ for creating new replica of shard shard1 for collection collection1
   [junit4]   2> 3569182 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_    x:collection1_shard1_replica_n21] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3570203 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
   [junit4]   2> 3570230 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema [collection1_shard1_replica_n21] Schema name=test
   [junit4]   2> 3570423 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 3570462 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n21' using configuration from collection collection1, trusted=true
   [junit4]   2> 3570462 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n21' (registry 'solr.core.collection1.shard1.replica_n21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39
   [junit4]   2> 3570463 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [[collection1_shard1_replica_n21] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/shard-1-001/cores/collection1_shard1_replica_n21], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/shard-1-001/cores/collection1_shard1_replica_n21/data/]
   [junit4]   2> 3570482 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@157089b
   [junit4]   2> 3570491 WARN  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3570585 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3570585 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3570586 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 30000ms; 
   [junit4]   2> 3570586 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 
   [junit4]   2> 3570587 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3570588 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.SolrIndexSearcher Opening [Searcher@7decc07a[collection1_shard1_replica_n21] main]
   [junit4]   2> 3570589 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3570589 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3570590 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3570590 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1619675212610011136
   [junit4]   2> 3570591 INFO  (searcherExecutor-12557-thread-1-processing-n:127.0.0.1:41368_ x:collection1_shard1_replica_n21 c:collection1 s:shard1) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [collection1_shard1_replica_n21] Registered new searcher Searcher@7decc07a[collection1_shard1_replica_n21] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3570608 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node22=0}, version=0}
   [junit4]   2> 3570610 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3570610 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3570610 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:41368/collection1_shard1_replica_n21/
   [junit4]   2> 3570610 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3570610 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy https://127.0.0.1:41368/collection1_shard1_replica_n21/ has no replicas
   [junit4]   2> 3570610 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3570621 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:41368/collection1_shard1_replica_n21/ shard1
   [junit4]   2> 3570772 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3570775 INFO  (qtp1772669569-133024) [n:127.0.0.1:41368_ c:collection1 s:shard1  x:collection1_shard1_replica_n21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1593
   [junit4]   2> 3570776 INFO  (qtp1772669569-133022) [n:127.0.0.1:41368_ c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:41368_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1609
   [junit4]   2> 3571044 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/shard-2-001 of type NRT
   [junit4]   2> 3571062 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_191-b12
   [junit4]   2> 3571063 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3571063 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3571063 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3571063 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@955b879{/,null,AVAILABLE}
   [junit4]   2> 3571064 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@21cf2967{SSL,[ssl, http/1.1]}{127.0.0.1:42426}
   [junit4]   2> 3571065 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.e.j.s.Server Started @3571321ms
   [junit4]   2> 3571065 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=42426, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/shard-2-001/cores}
   [junit4]   2> 3571066 ERROR (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3571086 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3571086 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.6.0
   [junit4]   2> 3571086 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3571086 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3571086 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-12-12T19:24:11.682Z
   [junit4]   2> 3571087 INFO  (zkConnectionManagerCallback-67769-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3571088 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3571088 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.RestartWhileUpdatingTest_21858C97DB913823-001/shard-2-001/solr.xml
   [junit4]   2> 3571092 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 3571092 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 3571093 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@3b60ae39, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 3571171 INFO  (OverseerCollectionConfigSetProcessor-73825758250991621-127.0.0.1:44855_-n_0000000000) [n:127.0.0.1:44855_    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3571308 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35241/solr
   [junit4]   2> 3571309 INFO  (zkConnectionManagerCallback-67773-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3571311 INFO  (zkConnectionManagerCallback-67775-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3571316 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3571319 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 3571319 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42426_
   [junit4]   2> 3571328 INFO  (zkCallback-67732-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3571329 INFO  (zkCallback-67758-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3571329 INFO  (zkCallback-67739-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3571329 INFO  (zkCallback-67750-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3571333 INFO  (zkCallback-67765-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3571333 INFO  (zkCallback-67774-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3571372 INFO  (zkConnectionManagerCallback-67782-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3571373 INFO  (TEST-RestartWhileUpdatingTest.test-seed#[21858C97DB913823]) [    ] o.a.s.c.c.ZkStateReade

[...truncated too long message...]

s.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/lucene/top-level-ivy-settings.xml

resolve:

jar-checksums:
    [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/null1919309442
     [copy] Copying 241 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/null1919309442
   [delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/checkout/solr/null1919309442

check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] 	found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] 	found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] 	found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] 	found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] 	found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] 	found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 34ms :: artifacts dl 4ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   8   |   0   |   0   |   0   ||   8   |   0   |
	---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...

-jenkins-base:

BUILD SUCCESSFUL
Total time: 452 minutes 14 seconds
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath$34.hasMatch(FilePath.java:2678)
	at hudson.FilePath$34.invoke(FilePath.java:2557)
	at hudson.FilePath$34.invoke(FilePath.java:2547)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1036)
		at hudson.FilePath.act(FilePath.java:1025)
		at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
		at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
		at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
		at hudson.model.Build$BuildExecution.post2(Build.java:186)
		at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
		at hudson.model.Run.execute(Run.java:1819)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath.act(FilePath.java:1038)
	at hudson.FilePath.act(FilePath.java:1025)
	at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
	at hudson.model.Build$BuildExecution.post2(Build.java:186)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
	at hudson.model.Run.execute(Run.java:1819)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

Re: [JENKINS] Lucene-Solr-NightlyTests-7.6 - Build # 23 - Still Failing

Posted by Dawid Weiss <da...@gmail.com>.
Unless the logging system is writing to raw io descriptors (or is
initialized earlier than test rules, which I don't think it should),
you could also use the:

@TestRuleLimitSysouts.Limit(bytes = ....)

to limit the amount of logs to a sensibly high limit.

Dawid
On Tue, Dec 11, 2018 at 2:15 PM Steve Rowe <sa...@gmail.com> wrote:
>
> And again!  Out of disk space caused by Solr HDFS tests (HdfsCollectionsAPIDistributedZkTest, HdfsAutoAddReplicasIntegrationTest and MoveReplicaHDFSTest, from https://builds.apache.org/view/L/view/Lucene/job/Lucene-Solr-NightlyTests-7.x/400/consoleText ), this time on lucene2-us-west.apache.org.
>
> Before I cleaned up the workspace, I captured one of the .spill files so it can be examined - looks to me like test logs - here's the head:
>
> -----
> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimDistributedQueue_E143A2C308722825-001/init-core-data-001
> 18827 INFO  (SUITE-TestSimDistributedQueue-seed#[E143A2C308722825]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
> 19271 INFO  (SUITE-TestSimDistributedQueue-seed#[E143A2C308722825]-worker) [    ] o.e.j.u.log Logging initialized @19443ms to org.eclipse.jetty.util.log.Slf4jLog
> 19340 INFO  (SUITE-TestSimDistributedQueue-seed#[E143A2C308722825]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
> 19396 INFO  (SUITE-TestSimDistributedQueue-seed#[E143A2C308722825]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
> 19556 INFO  (TEST-TestSimDistributedQueue.testPeekElements-seed#[E143A2C308722825]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPeekElements
> -----
>
> I'll make a JIRA later today and post the .spill file there.
>
> I think if we set suite timeouts on these tests, their failure logs will be produced, rather than Jenkins killing the JVMs upon whole-job timeout, causing .spill/.events files to be created.
>
> Steve
>
> > On Dec 10, 2018, at 10:50 PM, Apache Jenkins Server <je...@builds.apache.org> wrote:
> >
> > Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.6/23/
> >
> > No tests ran.
> >
> > Build Log:
> > [...truncated 31 lines...]
> > ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> > org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> >       at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> >       at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> >       at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> >       at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> >       at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:134)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> >       at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> >       at hudson.remoting.Request$2.run(Request.java:369)
> >       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> >       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >       at java.lang.Thread.run(Thread.java:748)
> > org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> >       at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> >       at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> >       at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> >       at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> > Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
> >               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> >               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> >               at hudson.remoting.Channel.call(Channel.java:955)
> >               at hudson.FilePath.act(FilePath.java:1036)
> >               at hudson.FilePath.act(FilePath.java:1025)
> >               at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> >               at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> >               at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> >               at hudson.scm.SCM.checkout(SCM.java:504)
> >               at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> >               at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> >               at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> >               at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> >               at hudson.model.Run.execute(Run.java:1794)
> >               at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> >               at hudson.model.ResourceController.execute(ResourceController.java:97)
> >               at hudson.model.Executor.run(Executor.java:429)
> > Caused: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> >       at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:132)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> >       at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:134)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> >       at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> >       at hudson.remoting.Request$2.run(Request.java:369)
> >       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> >       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >       at java.lang.Thread.run(Thread.java:748)
> > Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 at hudson.remoting.Channel@50787be1:lucene2
> >       at hudson.FilePath.act(FilePath.java:1043)
> >       at hudson.FilePath.act(FilePath.java:1025)
> >       at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> >       at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> >       at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> >       at hudson.scm.SCM.checkout(SCM.java:504)
> >       at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> >       at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> >       at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> >       at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> >       at hudson.model.Run.execute(Run.java:1794)
> >       at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> >       at hudson.model.ResourceController.execute(ResourceController.java:97)
> >       at hudson.model.Executor.run(Executor.java:429)
> > Retrying after 10 seconds
> >> git rev-parse --is-inside-work-tree # timeout=10
> > Fetching changes from the remote Git repository
> >> git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
> > Fetching upstream changes from git://git.apache.org/lucene-solr.git
> >> git --version # timeout=10
> >> git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
> >> git rev-parse refs/remotes/origin/branch_7_6^{commit} # timeout=10
> >> git rev-parse refs/remotes/origin/origin/branch_7_6^{commit} # timeout=10
> > Checking out Revision 719cde97f84640faa1e3525690d262946571245f (refs/remotes/origin/branch_7_6)
> >> git config core.sparsecheckout # timeout=10
> >> git checkout -f 719cde97f84640faa1e3525690d262946571245f
> > Commit message: "LUCENE-8592: switch the corrupted sorted index to a 7.6 version (instead of 7x)."
> > Cleaning workspace
> >> git rev-parse --verify HEAD # timeout=10
> > Resetting working tree
> >> git reset --hard # timeout=10
> >> git clean -fdx # timeout=10
> > Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data
> > Updating http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
> > Workspace appear to be locked, so getting a fresh workspace
> > Cleaning local Directory test-data
> > Checking out http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
> > A         README.txt
> > ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> > org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> >       at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> >       at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> >       at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> >       at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> >       at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> >       at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> >       at hudson.remoting.Request$2.run(Request.java:369)
> >       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> >       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >       at java.lang.Thread.run(Thread.java:748)
> > org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> >       at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> >       at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> >       at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> >       at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> > Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
> >               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> >               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> >               at hudson.remoting.Channel.call(Channel.java:955)
> >               at hudson.FilePath.act(FilePath.java:1036)
> >               at hudson.FilePath.act(FilePath.java:1025)
> >               at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> >               at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> >               at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> >               at hudson.scm.SCM.checkout(SCM.java:504)
> >               at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> >               at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> >               at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> >               at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> >               at hudson.model.Run.execute(Run.java:1794)
> >               at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> >               at hudson.model.ResourceController.execute(ResourceController.java:97)
> >               at hudson.model.Executor.run(Executor.java:429)
> > Caused: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> >       at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:132)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> >       at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> >       at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> >       at hudson.remoting.Request$2.run(Request.java:369)
> >       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> >       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >       at java.lang.Thread.run(Thread.java:748)
> > Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 at hudson.remoting.Channel@50787be1:lucene2
> >       at hudson.FilePath.act(FilePath.java:1043)
> >       at hudson.FilePath.act(FilePath.java:1025)
> >       at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> >       at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> >       at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> >       at hudson.scm.SCM.checkout(SCM.java:504)
> >       at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> >       at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> >       at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> >       at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> >       at hudson.model.Run.execute(Run.java:1794)
> >       at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> >       at hudson.model.ResourceController.execute(ResourceController.java:97)
> >       at hudson.model.Executor.run(Executor.java:429)
> > Retrying after 10 seconds
> >> git rev-parse --is-inside-work-tree # timeout=10
> > Fetching changes from the remote Git repository
> >> git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
> > Fetching upstream changes from git://git.apache.org/lucene-solr.git
> >> git --version # timeout=10
> >> git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
> >> git rev-parse refs/remotes/origin/branch_7_6^{commit} # timeout=10
> >> git rev-parse refs/remotes/origin/origin/branch_7_6^{commit} # timeout=10
> > Checking out Revision 719cde97f84640faa1e3525690d262946571245f (refs/remotes/origin/branch_7_6)
> >> git config core.sparsecheckout # timeout=10
> >> git checkout -f 719cde97f84640faa1e3525690d262946571245f
> > Commit message: "LUCENE-8592: switch the corrupted sorted index to a 7.6 version (instead of 7x)."
> > Cleaning workspace
> >> git rev-parse --verify HEAD # timeout=10
> > Resetting working tree
> >> git reset --hard # timeout=10
> >> git clean -fdx # timeout=10
> > Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data
> > Updating http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
> > Workspace appear to be locked, so getting a fresh workspace
> > Cleaning local Directory test-data
> > Checking out http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
> > A         README.txt
> > ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> > org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> >       at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> >       at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> >       at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> >       at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> >       at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> >       at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> >       at hudson.remoting.Request$2.run(Request.java:369)
> >       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> >       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >       at java.lang.Thread.run(Thread.java:748)
> > org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> >       at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> >       at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> >       at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> >       at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> >       at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> >       at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> >       at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> >       at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> > Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
> >               at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> >               at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> >               at hudson.remoting.Channel.call(Channel.java:955)
> >               at hudson.FilePath.act(FilePath.java:1036)
> >               at hudson.FilePath.act(FilePath.java:1025)
> >               at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> >               at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> >               at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> >               at hudson.scm.SCM.checkout(SCM.java:504)
> >               at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> >               at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> >               at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> >               at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> >               at hudson.model.Run.execute(Run.java:1794)
> >               at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> >               at hudson.model.ResourceController.execute(ResourceController.java:97)
> >               at hudson.model.Executor.run(Executor.java:429)
> > Caused: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> >       at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:132)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> >       at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
> >       at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> >       at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> >       at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> >       at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> >       at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> >       at hudson.remoting.Request$2.run(Request.java:369)
> >       at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> >       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >       at java.lang.Thread.run(Thread.java:748)
> > Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 at hudson.remoting.Channel@50787be1:lucene2
> >       at hudson.FilePath.act(FilePath.java:1043)
> >       at hudson.FilePath.act(FilePath.java:1025)
> >       at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> >       at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> >       at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> >       at hudson.scm.SCM.checkout(SCM.java:504)
> >       at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> >       at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> >       at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> >       at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> >       at hudson.model.Run.execute(Run.java:1794)
> >       at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> >       at hudson.model.ResourceController.execute(ResourceController.java:97)
> >       at hudson.model.Executor.run(Executor.java:429)
> > Archiving artifacts
> > Recording test results
> > ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
> > Email was triggered for: Failure - Any
> > Sending email for trigger: Failure - Any
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-NightlyTests-7.6 - Build # 23 - Still Failing

Posted by Steve Rowe <sa...@gmail.com>.
And again!  Out of disk space caused by Solr HDFS tests (HdfsCollectionsAPIDistributedZkTest, HdfsAutoAddReplicasIntegrationTest and MoveReplicaHDFSTest, from https://builds.apache.org/view/L/view/Lucene/job/Lucene-Solr-NightlyTests-7.x/400/consoleText ), this time on lucene2-us-west.apache.org.

Before I cleaned up the workspace, I captured one of the .spill files so it can be examined - looks to me like test logs - here's the head:

-----
Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestSimDistributedQueue_E143A2C308722825-001/init-core-data-001
18827 INFO  (SUITE-TestSimDistributedQueue-seed#[E143A2C308722825]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
19271 INFO  (SUITE-TestSimDistributedQueue-seed#[E143A2C308722825]-worker) [    ] o.e.j.u.log Logging initialized @19443ms to org.eclipse.jetty.util.log.Slf4jLog
19340 INFO  (SUITE-TestSimDistributedQueue-seed#[E143A2C308722825]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
19396 INFO  (SUITE-TestSimDistributedQueue-seed#[E143A2C308722825]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
19556 INFO  (TEST-TestSimDistributedQueue.testPeekElements-seed#[E143A2C308722825]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testPeekElements
-----

I'll make a JIRA later today and post the .spill file there.

I think if we set suite timeouts on these tests, their failure logs will be produced, rather than Jenkins killing the JVMs upon whole-job timeout, causing .spill/.events files to be created.

Steve

> On Dec 10, 2018, at 10:50 PM, Apache Jenkins Server <je...@builds.apache.org> wrote:
> 
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.6/23/
> 
> No tests ran.
> 
> Build Log:
> [...truncated 31 lines...]
> ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> 	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> 	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> 	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> 	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> 	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:134)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 	at hudson.remoting.Request$2.run(Request.java:369)
> 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> 	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> 	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> 	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> 	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
> 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 		at hudson.remoting.Channel.call(Channel.java:955)
> 		at hudson.FilePath.act(FilePath.java:1036)
> 		at hudson.FilePath.act(FilePath.java:1025)
> 		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> 		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> 		at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> 		at hudson.scm.SCM.checkout(SCM.java:504)
> 		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 		at hudson.model.Run.execute(Run.java:1794)
> 		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 		at hudson.model.ResourceController.execute(ResourceController.java:97)
> 		at hudson.model.Executor.run(Executor.java:429)
> Caused: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> 	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:132)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> 	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:134)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 	at hudson.remoting.Request$2.run(Request.java:369)
> 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 at hudson.remoting.Channel@50787be1:lucene2
> 	at hudson.FilePath.act(FilePath.java:1043)
> 	at hudson.FilePath.act(FilePath.java:1025)
> 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> 	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> 	at hudson.scm.SCM.checkout(SCM.java:504)
> 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 	at hudson.model.Run.execute(Run.java:1794)
> 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 	at hudson.model.ResourceController.execute(ResourceController.java:97)
> 	at hudson.model.Executor.run(Executor.java:429)
> Retrying after 10 seconds
>> git rev-parse --is-inside-work-tree # timeout=10
> Fetching changes from the remote Git repository
>> git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
> Fetching upstream changes from git://git.apache.org/lucene-solr.git
>> git --version # timeout=10
>> git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
>> git rev-parse refs/remotes/origin/branch_7_6^{commit} # timeout=10
>> git rev-parse refs/remotes/origin/origin/branch_7_6^{commit} # timeout=10
> Checking out Revision 719cde97f84640faa1e3525690d262946571245f (refs/remotes/origin/branch_7_6)
>> git config core.sparsecheckout # timeout=10
>> git checkout -f 719cde97f84640faa1e3525690d262946571245f
> Commit message: "LUCENE-8592: switch the corrupted sorted index to a 7.6 version (instead of 7x)."
> Cleaning workspace
>> git rev-parse --verify HEAD # timeout=10
> Resetting working tree
>> git reset --hard # timeout=10
>> git clean -fdx # timeout=10
> Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data
> Updating http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
> Workspace appear to be locked, so getting a fresh workspace
> Cleaning local Directory test-data
> Checking out http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
> A         README.txt
> ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> 	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> 	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> 	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> 	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> 	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 	at hudson.remoting.Request$2.run(Request.java:369)
> 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> 	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> 	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> 	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> 	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
> 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 		at hudson.remoting.Channel.call(Channel.java:955)
> 		at hudson.FilePath.act(FilePath.java:1036)
> 		at hudson.FilePath.act(FilePath.java:1025)
> 		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> 		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> 		at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> 		at hudson.scm.SCM.checkout(SCM.java:504)
> 		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 		at hudson.model.Run.execute(Run.java:1794)
> 		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 		at hudson.model.ResourceController.execute(ResourceController.java:97)
> 		at hudson.model.Executor.run(Executor.java:429)
> Caused: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> 	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:132)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> 	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 	at hudson.remoting.Request$2.run(Request.java:369)
> 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 at hudson.remoting.Channel@50787be1:lucene2
> 	at hudson.FilePath.act(FilePath.java:1043)
> 	at hudson.FilePath.act(FilePath.java:1025)
> 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> 	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> 	at hudson.scm.SCM.checkout(SCM.java:504)
> 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 	at hudson.model.Run.execute(Run.java:1794)
> 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 	at hudson.model.ResourceController.execute(ResourceController.java:97)
> 	at hudson.model.Executor.run(Executor.java:429)
> Retrying after 10 seconds
>> git rev-parse --is-inside-work-tree # timeout=10
> Fetching changes from the remote Git repository
>> git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
> Fetching upstream changes from git://git.apache.org/lucene-solr.git
>> git --version # timeout=10
>> git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
>> git rev-parse refs/remotes/origin/branch_7_6^{commit} # timeout=10
>> git rev-parse refs/remotes/origin/origin/branch_7_6^{commit} # timeout=10
> Checking out Revision 719cde97f84640faa1e3525690d262946571245f (refs/remotes/origin/branch_7_6)
>> git config core.sparsecheckout # timeout=10
>> git checkout -f 719cde97f84640faa1e3525690d262946571245f
> Commit message: "LUCENE-8592: switch the corrupted sorted index to a 7.6 version (instead of 7x)."
> Cleaning workspace
>> git rev-parse --verify HEAD # timeout=10
> Resetting working tree
>> git reset --hard # timeout=10
>> git clean -fdx # timeout=10
> Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data
> Updating http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
> Workspace appear to be locked, so getting a fresh workspace
> Cleaning local Directory test-data
> Checking out http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
> A         README.txt
> ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> 	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> 	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> 	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> 	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> 	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 	at hudson.remoting.Request$2.run(Request.java:369)
> 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
> 	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
> 	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
> 	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
> 	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
> 	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
> 	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
> 	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
> 	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
> Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
> 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 		at hudson.remoting.Channel.call(Channel.java:955)
> 		at hudson.FilePath.act(FilePath.java:1036)
> 		at hudson.FilePath.act(FilePath.java:1025)
> 		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> 		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> 		at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> 		at hudson.scm.SCM.checkout(SCM.java:504)
> 		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 		at hudson.model.Run.execute(Run.java:1794)
> 		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 		at hudson.model.ResourceController.execute(ResourceController.java:97)
> 		at hudson.model.Executor.run(Executor.java:429)
> Caused: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
> 	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:132)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
> 	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
> 	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
> 	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
> 	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
> 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 	at hudson.remoting.Request$2.run(Request.java:369)
> 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 at hudson.remoting.Channel@50787be1:lucene2
> 	at hudson.FilePath.act(FilePath.java:1043)
> 	at hudson.FilePath.act(FilePath.java:1025)
> 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
> 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
> 	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
> 	at hudson.scm.SCM.checkout(SCM.java:504)
> 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 	at hudson.model.Run.execute(Run.java:1794)
> 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 	at hudson.model.ResourceController.execute(ResourceController.java:97)
> 	at hudson.model.Executor.run(Executor.java:429)
> Archiving artifacts
> Recording test results
> ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-NightlyTests-7.6 - Build # 23 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.6/23/

No tests ran.

Build Log:
[...truncated 31 lines...]
ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:134)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1036)
		at hudson.FilePath.act(FilePath.java:1025)
		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
		at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1794)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
Caused: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:132)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:134)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 at hudson.remoting.Channel@50787be1:lucene2
	at hudson.FilePath.act(FilePath.java:1043)
	at hudson.FilePath.act(FilePath.java:1025)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
	at hudson.scm.SCM.checkout(SCM.java:504)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1794)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
Fetching upstream changes from git://git.apache.org/lucene-solr.git
 > git --version # timeout=10
 > git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/branch_7_6^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/branch_7_6^{commit} # timeout=10
Checking out Revision 719cde97f84640faa1e3525690d262946571245f (refs/remotes/origin/branch_7_6)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 719cde97f84640faa1e3525690d262946571245f
Commit message: "LUCENE-8592: switch the corrupted sorted index to a 7.6 version (instead of 7x)."
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data
Updating http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
Workspace appear to be locked, so getting a fresh workspace
Cleaning local Directory test-data
Checking out http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
A         README.txt
ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1036)
		at hudson.FilePath.act(FilePath.java:1025)
		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
		at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1794)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
Caused: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:132)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 at hudson.remoting.Channel@50787be1:lucene2
	at hudson.FilePath.act(FilePath.java:1043)
	at hudson.FilePath.act(FilePath.java:1025)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
	at hudson.scm.SCM.checkout(SCM.java:504)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1794)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # timeout=10
Fetching upstream changes from git://git.apache.org/lucene-solr.git
 > git --version # timeout=10
 > git fetch --tags --progress git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/branch_7_6^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/branch_7_6^{commit} # timeout=10
Checking out Revision 719cde97f84640faa1e3525690d262946571245f (refs/remotes/origin/branch_7_6)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 719cde97f84640faa1e3525690d262946571245f
Commit message: "LUCENE-8592: switch the corrupted sorted index to a 7.6 version (instead of 7x)."
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data
Updating http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
Workspace appear to be locked, so getting a fresh workspace
Cleaning local Directory test-data
Checking out http://svn.apache.org/repos/asf/lucene/test-data at revision '2018-12-11T06:16:40.379 +0000'
A         README.txt
ERROR: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
org.tmatesoft.svn.core.SVNException: svn: E204899: No space left on device
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:897)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1349)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:847)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:263)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:877)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:26)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:11)
	at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:121)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to lucene2
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1036)
		at hudson.FilePath.act(FilePath.java:1025)
		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
		at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
		at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1794)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
Caused: java.io.IOException: Failed to check out http://svn.apache.org/repos/asf/lucene/test-data
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:132)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:187)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1041)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1017)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:990)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6 at hudson.remoting.Channel@50787be1:lucene2
	at hudson.FilePath.act(FilePath.java:1043)
	at hudson.FilePath.act(FilePath.java:1025)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:937)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:864)
	at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:143)
	at hudson.scm.SCM.checkout(SCM.java:504)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1794)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Archiving artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any